Hello John,
" I don’t expect anyone to know the answer, but I’m hoping someone can tell
me what additional diagnostics might be enabled in the client that would
show what the client is waiting on? "
I think the suggestions of
*Justin Bertram* - Wednesday, February 21, 2024 9:56:28 PM GMT+1
are a very good answer to your question.
*Distinti Saluti / *Kind Regards
M.G.
Il giorno ven 23 feb 2024 alle ore 14:39 John Lilley
<john.lil...@redpointglobal.com.invalid> ha scritto:
> Vilius,
>
>
>
> Thanks for the reply!
>
>
>
> We have not seen this outside of K8S, but the question is why…
>
> We are hard-killing the active broker (shell into the node and do a “sudo
> kill -9”)
>
>
>
> And I can see in our logs that the backup broker senses the demise of the
> primary broker and takes over within two seconds:
>
>
>
> 2024-02-16 22:42:56,523 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
>
>
>
> AND it immediately processes other messages , which we log in the broker
> pod using a plugin:
>
> 2024-02-16 22:42:58,074 INFO
> [net.redpoint.rpdm.artemis_logger.RpdmArtemisLogger] SEND: …
>
> 2024-02-16 22:42:58,254 INFO
> [net.redpoint.rpdm.artemis_logger.RpdmArtemisLogger] DELIVER:…
>
>
>
> To be clear, ALMOST all of the failover succeeds within a couple seconds,
> and a lot of messages are being processed. But there seems to be one
> message caught in the act of sending just as the failover happens, and it
> takes 40 seconds to report an issue:
>
>
>
> 2024-02-16T22:43:35.988 [http-nio-9910-exec-9] JmsProducerPool.send_:376
> [9v2zwvclclrc] WARN - Error sending message, will retry
> javax.jms.JMSException: AMQ219014: Timed out after waiting 10000 ms for
> response when sending packet 71
>
>
>
> Our own application-level logging show that we attempted to send the
> message just at the time of failover:
>
> 2024-02-16T22:42:55.981 [http-nio-9910-exec-9]
> JmsRpcClientChannel.prepareCall:84 [9v2zwvclclrc] INFO - REQUEST OUT: {
> "header": {"version":1,"type":"get_task_status_request","id":"9v2zwvclclrc"
>
>
>
> We know this is the same message because we tag them all with unique IDs,
> in this case “9v2zwvclclrc”, and stuff that into a thread context for
> Log4j2 so it’s tracked.
>
>
>
> So our question is, what happened during the 40 seconds? Clearly the
> broker went down, and the backup sensed that and took over, and everyone
> quickly switched to the backup. Except for this one message. This is
> consistent behavior under load.
>
>
>
> I don’t expect anyone to know the answer, but I’m hoping someone can tell
> me what additional diagnostics might be enabled in the client that would
> show what the client is waiting on?
>
>
>
> Thanks
>
> john
>
>
>
>
>
>
> [image: rg]
> <https://urldefense.com/v3/__https://www.redpointglobal.com/__;!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3H50EbXg$>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com
>
> *From:* Vilius Šumskas <vilius.sums...@rivile.lt>
> *Sent:* Wednesday, February 21, 2024 2:44 PM
> *To:* users@activemq.apache.org
> *Subject:* RE: HA failover: Nothing we try reduces client recovery below
> one minute
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Hi,
>
>
>
> are you able to reproduce your issue without Kubernetes layer on Artemis
> instances? I’m not sure how exactly do you kill a pod, but 40 second
> timeout very much looks like default pod grace timeout + 10 seconds.
>
>
>
> --
>
> Vilius
>
>
>
> *From:* John Lilley <john.lil...@redpointglobal.com.INVALID>
> *Sent:* Wednesday, February 21, 2024 8:45 PM
> *To:* users@activemq.apache.org
> *Cc:* Lino Pereira <lino.pere...@redpointglobal.com>
> *Subject:* HA failover: Nothing we try reduces client recovery below one
> minute
>
>
>
> Greetings!
>
>
>
> We are having a devil of a time trying to reduce delay during a failover
> event. We’ve set our URL to
>
>
> (tcp://dm-activemq-live-svc:61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=200&initialConnectAttempts=200&clientFailureCheckPeriod=10000&connectionTTL=10000&callTimeout=10000
>
>
>
> But nothing seems to reduce the time it takes for a sender to detect the
> issue and failover. With or without these parameters, it takes a full
> minute to recover after failover. Are there other parameters to adjust?
> Something in the broker? Is there some internal retry loop and timer that
> waits longer that we can influence?
>
>
>
> We are testing failover by killing one of the AMQ pods. This almost
> always succeeds without issue, except occasionally it doesn’t.
>
>
>
> The backup broker log shows that it assumes control very quickly
>
> 2024-02-16 22:42:56,523 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
>
> 2024-02-16 22:42:56,533 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61617 for protocols
> [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
>
> 2024-02-16 22:42:58,074 INFO
> [net.redpoint.rpdm.artemis_logger.RpdmArtemisLogger] SEND: HEADER=
> {"version":1,"type":"void","id":"jagbjffeu3of","api":"hpa_metrics","method":"get_expected_pod_counts","authorization":"Q979k6lzOu9KmqMA89GbvmZnIAMNpJJP/TEAAa7Yjpo="},
> BODY={"message_type":"void"}
>
> 2024-02-16 22:42:58,254 INFO
> [net.redpoint.rpdm.artemis_logger.RpdmArtemisLogger] DELIVER: HEADER=
> {"version":1,"type":"void","id":"jagbjffeu3of","api":"hpa_metrics","method":"get_expected_pod_counts","authorization":"Q979k6lzOu9KmqMA89GbvmZnIAMNpJJP/TEAAa7Yjpo="},
> BODY={"message_type":"void"}
>
>
>
> In our app logs, we show that we are attempting to send a message to the
> AMQ broker at the time of failover:
>
> 2024-02-16T22:42:55.981 [http-nio-9910-exec-9]
> JmsRpcClientChannel.prepareCall:84 [9v2zwvclclrc] INFO - REQUEST OUT: {
> "header":
> {"version":1,"type":"get_task_status_request","id":"9v2zwvclclrc","api":"test_harness","method":"get_task_status","instance":"combined","authorization":"***REDACTED***"},
> "body":{"id":"d1123865-ac47-4238-ab02-5f2324a43264","progress_start_index":0,"message_type":"get_task_status_request"}
> }
>
>
>
> But *then it takes 40 seconds for the “10000ms” timeout to happen.*
>
> 2024-02-16T22:43:35.988 [http-nio-9910-exec-9] JmsProducerPool.send_:376
> [9v2zwvclclrc] WARN - Error sending message, will retry
> javax.jms.JMSException: AMQ219014: Timed out after waiting 10000 ms for
> response when sending packet 71
>
>
>
> Our problem is really that this 40-second delay seems to push everything
> back so that recovery takes over a minute. Once we get into the one-minute
> mark, we start hitting several timeouts, like our own RPC timeout setting
> and the nginx ingress controller for K8S. How can we know why this takes
> so long? We’ve set every timeout we can find for the AMQ client to 10
> seconds. Is there some other setting we need to adjust?
>
>
>
>
>
> Meanwhile the primary broker pod has returned, and the backup decides it
> is not needed:
>
> 2024-02-16 22:43:16,115 INFO [org.apache.activemq.artemis.core.server]
> AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.31.2
> [10952195-b6ec-11ee-9c87-aa03cb64206a] stopped, uptime 16 minutes
>
> 2024-02-16 22:43:16,115 INFO [org.apache.activemq.artemis.core.server]
> AMQ221039: Restarting as Replicating backup server after live restart
>
>
>
> But our app is waiting for its response on the reply-to queue, and gets
> this error nearly a minute later:
>
> 2024-02-16T22:43:58.033 [Thread-6] JmsStaticConnectionPool.onException:78
> [] ERROR - Receive error occurred. javax.jms.JMSException:
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
>
>
> This is the stack trace at time of the first timeout:
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(
> ChannelImpl.java:550
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fChannelImpl.java*3a550&c=E,1,hb7aGgx-vHCuvCZitz5EiH3vZi_XC1uzDAFkEHTxXekOnlXOXMP4ibOn6V4KDEB1MD9djxVG0pCPgN9wOUF6dYSlvQjZDxnFdtoXzY5E9EG2FGcoIhbpnw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1g6FXp6A$>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(
> ChannelImpl.java:452
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fChannelImpl.java*3a452&c=E,1,gF3UMbAQeYfSPT8cA2REDOjwMzwGDYxipRXgXQCDiWReJE7I74vBcqvEKJaemOXlOU1-G4hJB4qNbaM1K_LmdumKZ7DAKdaqBEqDSniSuD46bi8,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt20cJ_Apg$>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(
> ActiveMQSessionContext.java:588
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fActiveMQSessionContext.java*3a588&c=E,1,9Q1r0SpyXwLIB-EMPBJw-aI7zdjFiqijbOY_h2LMwTn-B3FR_vV-GEiHQ8QWZkRub6NnY7OQvVJRoEEjiWottZjxkgKMpVcwv7wPDZiq6s8LgQ,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3h25NFSw$>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(
> ClientProducerImpl.java:305
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fClientProducerImpl.java*3a305&c=E,1,_M2sIEUuJWPDvj5jVjnOj7fletSz95vJyLou8Gve4mp0Wae3oDPYH9uOyeCWbMIwsiPMoIiEo_WDB9yfskeia55SN4MiEe3LZEGXVrtUFone1wdABZb6gOPC2m9w&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0SFxm2mw$>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(
> ClientProducerImpl.java:277
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fClientProducerImpl.java*3a277&c=E,1,oQ1uB8jzdsj7gUjIdMAYjGGFhBP2jGP0m_2VkTenJuUFc4-qsEa92fBmhf4b0zrZRLdm-Ctl5FiTaRL7ALhu7_f_P5fL9RmiP1TcxNc7wZhHCcA,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2SJGiapg$>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(
> ClientProducerImpl.java:147
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fClientProducerImpl.java*3a147&c=E,1,2JXRlcwBE1DgWg6Nk_YIcpOYWcBvJBBdvO075qubJv8IiA1-t8CMakQozpCrfoZ0t4l0nLmaIhS5d6py-XVc-yhJzsTx-N42mWmNZOT6yHDC2ljCuzo,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2KXjWXmg$>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(
> ClientProducerImpl.java:129
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fClientProducerImpl.java*3a129&c=E,1,Jzd6hLneXOmyA5D7vxY21LhDyWMJrnaXXxf8o-gF4Erwuf8q-9gnnJY-qV8bRHP7bHdZZtiUFhv1qNRCQl-ysyzGr6_YjHKyBoOjv1rEjOYdhzI0zgkpOw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0vpIbCDw$>
> )
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(
> ActiveMQMessageProducer.java:483
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fActiveMQMessageProducer.java*3a483&c=E,1,0yivDeiDyAHDMF-bwOucG30LADmXrdItj_pQJnK_SRY9vGr_DfKM_LL-LEoVVOZ6x34Zuzto8nyL_HNWC7xmqAe7ZEjcFKAUJSZ-Yjv_eFT-PJ-2pUKa&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt34eeiSRw$>
> )
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(
> ActiveMQMessageProducer.java:221
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fActiveMQMessageProducer.java*3a221&c=E,1,NuJzonWY3dCHvQuGAvWh5R9wQe_JiLaD6RfhoD0drObSHoBwRrh5zqwLDiOv2F3MUqtErz3Q5t8JJj5iqy9DHLusThoeoJwww3tnwamhrMVFj1oZ7aSh5Q,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2iAw6dFw$>
> )
>
> at net.redpoint.ipc.jms.JmsProducerPool.send_(
> JmsProducerPool.java:372
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fJmsProducerPool.java*3a372&c=E,1,fLwu6-XhE-35j29l5My769fvdrx1oKFsEumlcnxIJ-zl3lpqQHIknDa4dZS2DSKoRlr4XIkh18khm9yTNif1ZUgCfPW4T3JUc54rnFzytNc,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3YTy5riw$>
> )
>
> at net.redpoint.ipc.jms.JmsProducerPool.sendRequest(
> JmsProducerPool.java:301
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fJmsProducerPool.java*3a301&c=E,1,L5JFmpnTDTQel8SbnG3VK4xEP8oY_gtJYyoVXO_F1DJnNe5rrOo4AySDLqTxMtiqc5Sv2j-a2dIG16jBZ-d4LNnUK_3MUhcD9diVP_2Do4_KTe9r&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0tq8gcfg$>
> )
>
> at net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(
> JmsRpcClientChannel.java:228
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fJmsRpcClientChannel.java*3a228&c=E,1,CZrfHT-US8E2FAm8Os73ubg9knljLI7YUiud8KYRU6cShYCwqNJ5cz2B2aC_G4nBOGULRt1AZed5AhfU9Qu2pubWVzKotlmqo6chQELiwyc4k4BjKvy0702Q&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3_4x6g6w$>
> )
>
> at net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(
> JmsRpcClientChannel.java:202
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fJmsRpcClientChannel.java*3a202&c=E,1,gv6BduvhcXnCEEmt3MhN4xVBzok9tDgp18LcXG5zlG49w7C_x40T8KqCdYngblOJh4unBVqWeWqv_kwjWTHbFq4jos3V8aInP1_BWc6Q&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2s59WSyA$>
> )
>
> at net.redpoint.ipc.jms.JmsRpcClientChannel.call
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fnet.redpoint.ipc.jms.JmsRpcClientChannel.call&c=E,1,nR8Axi6nKaDe0idP2FZMm4-S0OB2WBHH0-vP-7T0n4y8bP2jnr8PyKQEkIh-GNbOulFVyo-JmkNEsFJq2kRASqXXzC1nK5ih-T8jawiX12EX18KZ&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1_dQaRyQ$>
> (JmsRpcClientChannel.java:101
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fJmsRpcClientChannel.java*3a101&c=E,1,uDLLFYMJiricH0b2CCzCbONAYihnNI1rkLL_9o3o5XX8VSwfL42BSuXV314BKJbKYXk3dquUadeP3jwSJj-tPB3mO2FYtaFj3fh_2x6yXBK-Qtar70K_&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0erUTSeg$>
> )
>
> at net.redpoint.ipc.clients.RpcClientBase._sync(
> RpcClientBase.java:169
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fRpcClientBase.java*3a169&c=E,1,wyTB_YLwnilukCP4yaV-KnfiFMrJ_d219DnfTQjBnr0T8xdChqgpha2b3O0CcQKmua6RRShv7kQXt2cDL57kooPnGPR3-BRqa6PUUlyTB7iV2Lv0z7njICo,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0D2cXUuQ$>
> )
>
> at net.redpoint.ipc.clients.RpcClientBase._rpc(
> RpcClientBase.java:237
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fRpcClientBase.java*3a237&c=E,1,LWdl8WVF2wTGJOIp7_iMum6NPwuAg1IuIskb-1ta9DuBVPvU2ZUDiHcWfbamo5braSd7dxBcn43KYr_foQZvO0TXasHJKFRnAzVHKKivXATR_x2T&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3URjlVSQ$>
> )
>
> at
> net.redpoint.rpdm.ipc.clients.TestHarnessClient.getTaskStatus(
> TestHarnessClient.java:229
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fTestHarnessClient.java*3a229&c=E,1,PgvRlByMcnxeUXiBba7IiePoAnF4IVs_7Lxnrd_8jq8Q-fIuuubZr_y7olwgeULE3XlvUsXDNp3_9NhOFKteaq-i16oXJtIJx_X7AEnPucP-fu4,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2AY4kxNA$>
> )
>
> at
> net.redpoint.rpdm.ipc.web_service_gateway.TestHarnessHttpServer.lambda$getTaskStatus$25(
> TestHarnessHttpServer.java:390
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fTestHarnessHttpServer.java*3a390&c=E,1,6IBjNSLNijOWzvN4rhSDmgiqTjpEHPembkvVBDdhfGLziM1KCtYXTrNIXtmV93l3zOoKEVXSeigVB8bRDpPEy6lhIMT7eXHSfG3-EgPqV8YVibSm3GK7URjjPw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2GZtq9pA$>
> )
>
> at net.redpoint.ipc.SecurityControl.doAsNoThrow(
> SecurityControl.java:272
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fSecurityControl.java*3a272&c=E,1,JEaIr0zNLFXmv2-s5Q2nqgJtrQ6Xe_7nI1J5cgFtpjXbGHFfuEDxpXCJql-hCKG2zwv67dnsNsq1bDiCfHaGOM_1-um8PTmJwWr5lly2wBvaCDnxCP7xLoW6&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2pX8cpDQ$>
> )
>
> at
> net.redpoint.rpdm.ipc.web_service_gateway.TestHarnessHttpServer.lambda$getTaskStatus$26(
> TestHarnessHttpServer.java:390
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fTestHarnessHttpServer.java*3a390&c=E,1,cr6EzskPARQs3atjS-qqk13h29wSx5_Ost_yPOnOsCvrReemnvm5SneMupe5u7j3QG-MQNRzjiHhaPSCz14D0aS4qcC8AqMjGPxxr5ldyf0,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0bMUq77g$>
> )
>
> at net.redpoint.ipc.InstanceControl.doAsNoThrow(
> InstanceControl.java:84
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fInstanceControl.java*3a84&c=E,1,KEjKjtppW9ddjt2brqtRGzD7gf9hKSZaBiRK_pT4nfQlFKVX7I1MI6xmv7Ae7Haq7lU1mm5AEvHVdurLVtx75mtR8WhQ2DpUekgrYqPCFIV_&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1JIG_JIw$>
> )
>
> at
> net.redpoint.rpdm.ipc.web_service_gateway.TestHarnessHttpServer.getTaskStatus(
> TestHarnessHttpServer.java:390
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fTestHarnessHttpServer.java*3a390&c=E,1,xxxxXpDzVQJIHIDnITNGO8paqNClGREjDResRK_0fPJ4RlIbFS0gq33a6q6Fa_ytIrAq7LiutsX_07eAsBrSW1D8OSUlQaj0eJP6XFt691Pv&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2w7GGphQ$>
> )
>
> at
> jdk.internal.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fDelegatingMethodAccessorImpl.java*3a43&c=E,1,dtPs-F3WSSDUIYKORhz5rbXvE7bGcxPI4Al1JWQIRY8xj66Q7Ack4YUCB8Nw7udnKiMx1n-rq-No4Nyj3whIzuGpiFBSboAGz7zyfyMa5g1qE_BQwIFkFd4m5g,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3mKGevRw$>
> )
>
> at java.base/java.lang.reflect.Method.invoke(
> Method.java:568
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fMethod.java*3a568&c=E,1,_UdYZqtoNh8X1NEr2kGR0aTolwK_yJVCjNFBA-hXoLQT1e43ZfwtMHdJIUIGbzyINUpdy8FRnzLB2ii_7KLn6WR7Hxl6bLhJEUOb5-ewhU2IJkgKbQ,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3pH5PkTQ$>
> )
>
> at
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(
> ResourceMethodInvocationHandlerFactory.java:52
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fResourceMethodInvocationHandlerFactory.java*3a52&c=E,1,_ANSTEmLreJLEQz9RwIa3Hq0m-XWtYh1mmCLvHtKh_Nl94vWQgtz8kzCx6Su-0NUwC_E5f9R0_G34M1fcvyw_tbwt2BabAq1ByTCwEGQyOUk&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3Bot_Jpw$>
> )
>
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$
> 1.run
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2f1.run&c=E,1,MZVqn6IzVn_wW73NZf9lIBKoDm6cMaU1WgNNYImFDQDDEY5Z0iFVnAvgt7x5mIfTkUF09OXzo9CFw-vPVsul-Izp5qW2Yrck4_6eujcuN-u_sllrUoywmJU,&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0SmMpshA$>
> (AbstractJavaResourceMethodDispatcher.java:146
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fAbstractJavaResourceMethodDispatcher.java*3a146&c=E,1,kizlN5vIvVLLgA2heqHvJlSyoVlUscXya5obSG66VCdZ-DUNMo10tEZwrh5x-1BykW4pb2xn-vEOY43NbjHHVG1y_FwY-MMYotiVd428HZlx9w,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt16EunjTg$>
> )
>
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(
> AbstractJavaResourceMethodDispatcher.java:189
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fAbstractJavaResourceMethodDispatcher.java*3a189&c=E,1,u_xqzYFUz2Mx9lP8JBBlRUcq32BdYTzMnt7peBRr2FpOwDFlvp4liz3EBNUbZq1ZLE9BfvsVlloDJ5TIxFdivqlPSdCZc1dzcixUXMHvni4,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt20wlRH3Q$>
> )
>
> at
> org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(
> JavaResourceMethodDispatcherProvider.java:219
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fJavaResourceMethodDispatcherProvider.java*3a219&c=E,1,W8GCe6oN-l2zCyZhE5onAlq0kIKg8WCIQAKI7Ert2GHO2-GAaKAyzEWTkO68IsniGdhLmuMMq47ojjdzidPYom1g5MXCCxZ6w8ZtJ29AxhR0Pw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt01pruuyg$>
> )
>
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(
> AbstractJavaResourceMethodDispatcher.java:93
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fAbstractJavaResourceMethodDispatcher.java*3a93&c=E,1,vnLQ__mSY8cdDiqo3_dWQgOEaspcJJhuBzEQ4DMznsTMKXqZP1lx2dcwqn2aSOQqSY8YEfuBsU4WYUXtAQVpglh1dQ9kz-Gn-SLO_r4D9zMt&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3IIxYpxg$>
> )
>
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(
> ResourceMethodInvoker.java:478
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fResourceMethodInvoker.java*3a478&c=E,1,vtRvnVh3vxShrzucCuAx-7fWKPkzGkUiH9DjAnxttNhMJhqObrBBWGcDx9l5cfjoZoQ6vR2TNiDaql1bs5uP0jK4Qo4Y9VbjuILqGjYTKFNaZsJYT9pV9Z_FqA,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1zJfVL4g$>
> )
>
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(
> ResourceMethodInvoker.java:400
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fResourceMethodInvoker.java*3a400&c=E,1,QbIG4TxOeShedV6j671Uv5vLnReJwXRCbSGagUyectQ5EHwxr5ovmRa6WhLDZWqioys1UYyT-20IpKMg6KQ2o1XXUkcy2oGZeW4p_KmXBBE,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2Z-QVu8g$>
> )
>
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(
> ResourceMethodInvoker.java:81
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fResourceMethodInvoker.java*3a81&c=E,1,ZVyXYuf1u2G26rwo3XVG3Z8_2or4DcafcB3zlO2SdB6wb6R4w11zwbOHrLj_ZPmhrx3O6_AnSfWzWJBGi3JDQzxGByxzRTNjKqm71joHVWUdQXigItfn9Fpw&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2zc6H2DA$>
> )
>
> at org.glassfish.jersey.server.ServerRuntime$1.run
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2f1.run&c=E,1,5VdU5X2PgA-c6MijT2k3BoWYlOm_QZvwG-1u3s1bvib1G3MgAOipXud40Vx8MwfBxJLGeYU4FhLDJ59Nv51dBb3lxaCq0-DairoRJsZt&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3uMp13Og$>
> (ServerRuntime.java:261
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fServerRuntime.java*3a261&c=E,1,ngWN53pTjMa1KEncK0_jy2Mi8gO-yrsz4XmCGEgfdXTCBWl7AyOAIgrwcmQjj3uDCpIWPfc05kbIwQFDlzeALGQA2c6p1LRwHvO9XSw13_JOQ_1tpN1NMthFbg,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3Irprhkw$>
> )
>
> at org.glassfish.jersey.internal.Errors$1.call
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2f1.call&c=E,1,gpcxIIgn7_ZegqcuKnXoAFvZ31ppjxUZnNewxEWON2yyskEr-VLMK-VKPnUhIvj-jg6fw-FemOr_TjszVBXOK8O8Z9sysAKjrd58HGYpgRy8HhVUKKTiGziR358,&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1BpwtCqA$>
> (Errors.java:248
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fErrors.java*3a248&c=E,1,-4lNjwYENaRsriy3ayEWHiwSRHlsRk0A2JKqmAIjx8UYUguXhgukOJmIcAe7K-6Ug4fPj2aSGnE6qK8XTSl9214Wgyrnumxj1RQZLtMncSxqqpZkM9ANRg7g&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2FA6Qtiw$>
> )
>
> at org.glassfish.jersey.internal.Errors$1.call
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2f1.call&c=E,1,13JGq-_iEwBcwV5mPfYx3V7i_FkPJ4YflLbmNmIIFHY0c-VPpXwqgbZEC59xZM7gjk84I1s10Wo1esMdauVsqwj6k90skMDADwp_MdPvwNupS2jKtlQ9Lw,,&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt34m4sJiw$>
> (Errors.java:244
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fErrors.java*3a244&c=E,1,DeHOvMPeAsDyExSsCn1tzVYIUwljHqkTLT2fVzPpTOvt2eUNGvdIvGN0zuwdIKxOq0vcLo9gG7Q1rklzEELzIllI78LKvuVzlAWC2mda--WK7Bd5RgbBybvv1w,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0ZA_QU3w$>
> )
>
> at org.glassfish.jersey.internal.Errors.process(
> Errors.java:292
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fErrors.java*3a292&c=E,1,-2TodtPTpE03ZGg_ww8ftKGn_pNdZxKFDN9YlA6us-U8mFuIRhb8cyqlQ46PSABDEdtcVCgQQ6p9DWK7goIr6ihIifN518cQeV2vLZuaQ14I&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2zpyDq2g$>
> )
>
> at org.glassfish.jersey.internal.Errors.process(
> Errors.java:274
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fErrors.java*3a274&c=E,1,c1aDRBx-Kkhe46k62MbX3RYaKICs6JicYQPPJa24Veuptu1ic45DvtPjKO3vn_lsZcG6cBgBB-DB-mG24cbdHPoUqPGioK3RpW5rE9QWCjRdl8yLPVgPCHgMxA,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2N07l4cA$>
> )
>
> at org.glassfish.jersey.internal.Errors.process(
> Errors.java:244
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fErrors.java*3a244&c=E,1,w1FDKjTa9Six6ZVu9g3QJYnGQuflyNbum7nYq3utYVnzfWFOiu1z4auY76BL3tgpquZCZRb3cWcZ2UWireyX03yjfuZVK1yq5_8_qi1GwLstFZa1Q0dcxQ,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2Hw2nqzg$>
> )
>
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(
> RequestScope.java:265
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fRequestScope.java*3a265&c=E,1,TgExyJ7UXeTRjHTUHoKEFcexeWKopISAsK-5WCliWSvgVasuQhW_xB_ZKivdXmCAR9HosV8xJIfkTyvZaKzQoxbnPgNoKkQwAWteG-9-AuEf4w1h60U,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2QswbnQw$>
> )
>
> at org.glassfish.jersey.server.ServerRuntime.process(
> ServerRuntime.java:240
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fServerRuntime.java*3a240&c=E,1,I-aQ9e78vqVvBnz_Ul46LjEl5AxbyOb5qBkDMavndcJMjZBTPPCbrCTBUt-UYdXaoOZ4u29XzqDfxy2eTamMsdzEfg44HfbufG4C3oM5kbFn78SgQA,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3Ju84Unw$>
> )
>
> at org.glassfish.jersey.server.ApplicationHandler.handle(
> ApplicationHandler.java:697
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fApplicationHandler.java*3a697&c=E,1,Jsego7u0OUnIQqzGf5sprW5iJZCmr2L2KDCQOBs43_3NIuQXU9RPUgPHy-bG-YKQL7wVaDTLxoHhUOB58RkM-ybq2AWCr5aWqmR6B6Ob&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0E3JvDwg$>
> )
>
> at org.glassfish.jersey.servlet.WebComponent.serviceImpl(
> WebComponent.java:394
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fWebComponent.java*3a394&c=E,1,lTSTlVXOoWmj6glMwmyv2CE_mfCD0zE1J8-2LyhxmJBQqBSViWGsyg_vaBj2Vrjv3_7zkGzv2VPu8UIfg98HtikJNOYTpW62dTRkVdbMnC7pvnA5-B3C6Db6YQ,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3wrzSdjg$>
> )
>
> at org.glassfish.jersey.servlet.WebComponent.service(
> WebComponent.java:346
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fWebComponent.java*3a346&c=E,1,KOLlb6JskDrWev0dVrCArHDSpfP5fNCXtw_oCiOYEJlHIJfINTLNNeKhy-H1ArSiQ-0ENNaD_Qh0Geb8X7wqrqwfpqBwLtEm4G98wam8F7O3&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1NH6JNvQ$>
> )
>
> at org.glassfish.jersey.servlet.ServletContainer.service(
> ServletContainer.java:357
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fServletContainer.java*3a357&c=E,1,xE8kQtKULT9CbuOJSHjcAq8nmkXVmrwW0pi6-5d4-ohnPEEJFxn_EinkKIPz_1VrgyvWUsHXNZFDtupRmAsrQqhTRXENL7kSa5LIjuqNojj2cwdWi2oRKE17U9l1&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1pV22FEg$>
> )
>
> at org.glassfish.jersey.servlet.ServletContainer.service(
> ServletContainer.java:311
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fServletContainer.java*3a311&c=E,1,CGR56oCDqMxWtXmFVVxBImiJVtbbAnuo8i75aV-g9KkYVlBBWtMCnF1kTybr_6JDC2vBezhqfLbIBUgYve0GY6zk_eFI22NTS969ZV3KkquxqBkl_Lk547k,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1w8YLu2g$>
> )
>
> at org.glassfish.jersey.servlet.ServletContainer.service(
> ServletContainer.java:205
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fServletContainer.java*3a205&c=E,1,RHUk7TgmAHI59I04g3HWyoryTWNqfXzhI5EEKYB6gOr3gsQ78kYA1UJwYXbIx_4c8K7uM2fG6K1GtP3ejLMG3qYpXpr8gyK3hl736QY7FLSbJ7on3G0s9_4DBQ,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3iKuJlPQ$>
> )
>
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:205
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fApplicationFilterChain.java*3a205&c=E,1,h2cMVon6xGXZ_SOU3zyFb-JtijRHlxgdyfCJHNtTXWCsZudhQv2hsGK9Gq6da_M2mu0HnRL_qoXqRZzFaeGKW1zw9MTKaY8hl3QmPlydzgE,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt00e4waYA$>
> )
>
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(
> ApplicationFilterChain.java:149
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fApplicationFilterChain.java*3a149&c=E,1,19pKUon60z33K_WmMwfnNMVHc18KirhbBq6O1sGxhUvEzfCHQv-8kpmVCoTKdZjNpRBlG9i1L6hSsevfEW4g3oEQ2demPaHgOBz0STxPqH71RVjdOw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2wFFxX-Q$>
> )
>
> at org.apache.catalina.core.StandardWrapperValve.invoke(
> StandardWrapperValve.java:167
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fStandardWrapperValve.java*3a167&c=E,1,iurW6Vg5HqcJe6goPvo4sb_OPJZ1ug8vrXm4M5nm_3WfLnrKdoR6AzqfMjjhLJEBGN1YTuVSr2KGjz7aPIdHNMWq5LroYkYwN_9ERWrPAw0qj-0qjeY,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2UbL4H-g$>
> )
>
> at org.apache.catalina.core.StandardContextValve.invoke(
> StandardContextValve.java:90
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fStandardContextValve.java*3a90&c=E,1,y9KIkhbbvuOVtZ1BMVlp8LM8grRux9d-SB3Cx1ysNOkTTGAzWyKiaqYmfjtiyJ81e7xkDEvfQB5PBjpwh9D-qsgkZbLVC4OA7kBl9XhQ5D-wcaB8OGarKkgSpgs,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2aG4yB2A$>
> )
>
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(
> AuthenticatorBase.java:482
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fAuthenticatorBase.java*3a482&c=E,1,870FeHRgrrt8FiPhpmG_J2QY0UhrzX4l2obWvn7GSTBkSokbpyaEh7DA5wlIFPxx8U8UucJ5jxsveJBiGv73ZL23tRaxKmL0sqlnVeg93LTGYv0,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3ZiI4UDg$>
> )
>
> at org.apache.catalina.core.StandardHostValve.invoke(
> StandardHostValve.java:115
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fStandardHostValve.java*3a115&c=E,1,gtlhUbg7fRCSWrxRdn-xvp5rZiVVL14r6bmxhD8tbUZeO2HPLW42mnvDUaqWB165vffTw4pqzZBrvfcsJ0iwuZ0uL2tGE34M50Vn9JteanRmGYMeFJ4szU0g&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt06UD8C6Q$>
> )
>
> at org.apache.catalina.valves.ErrorReportValve.invoke(
> ErrorReportValve.java:93
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fErrorReportValve.java*3a93&c=E,1,8fP04ef08snHBtrmso4oiWrZWAQg8unZ-PhOZT6v_YO0Udv7Sno4UnmuKPE-yPpc-WPoAJoSYfC_Kep5_Sf1xeu9i9jJAiyeGn-90SXaPOdE6PJa3eejOnM,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0_U4I5Lg$>
> )
>
> at org.apache.catalina.core.StandardEngineValve.invoke(
> StandardEngineValve.java:74
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fStandardEngineValve.java*3a74&c=E,1,5XJYADhwE12w-qyHEEVFAnAJ_Fge20HRlvGrtNOUAjzT4UhVYhPeh7qsZCiR11GlnfnX7mxaIgwiug6752Gk5JURAIU1PgK6rk8vqsfvxM-7mbRJ&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2uszZYIA$>
> )
>
> at org.apache.catalina.connector.CoyoteAdapter.service(
> CoyoteAdapter.java:340
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fCoyoteAdapter.java*3a340&c=E,1,B5JOWKWHEb6cVQgZlyltFnVbK4WhfVdrczK7jov9-MUlkwYC0zpCzTBNnohFstlGatoq7jDST-m-NLHFTHYVTuCNbqPIrZ8ufFHamINtG3N05p9UlkMwd5oEwSc,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1Yco2SdQ$>
> )
>
> at org.apache.coyote.http11.Http11Processor.service(
> Http11Processor.java:391
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fHttp11Processor.java*3a391&c=E,1,iDXL8VhEKOyTibuCsANH061qEr0KkrGbiIOGzj_8fzqISA7HsdtnrEMfo09pTcpyQepntY3cgZKYSXPSK2eh3GO99UHkbTF4T7L5Z3IXl0NAyVjPSjm1ew,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2tgXW6Xg$>
> )
>
> at org.apache.coyote.AbstractProcessorLight.process(
> AbstractProcessorLight.java:63
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fAbstractProcessorLight.java*3a63&c=E,1,A-vRfgPyPTwRwif1XCFkrF1_a4zD_DegwPnekB4SOkm04OpYSrniO6S-gm-ixgSAkSgUkZja9NWO2MYTIRlh6eLiLl2YKnrecwml7gwidbw2WF7bOGQ0Vw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3EWG8C9w$>
> )
>
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(
> AbstractProtocol.java:896
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fAbstractProtocol.java*3a896&c=E,1,pDFoRtt82LZU-FLdcBDdk-qlTk-Kt6gnRQpSQzYU8dEYEOGLmK01S-_w3L8iQgukNx49f3AUgrw9O1VhBVlRfoH4PNsuKkvr7sPvOf1bPaE,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1FxxcYyw$>
> )
>
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(
> NioEndpoint.java:1744
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fNioEndpoint.java*3a1744&c=E,1,-7FPn1c3059u04P9uwPVOkko51iX8BMN0E_rPpTPXwhN9hkjEjkHEsthD60wkwjFhcfxkwdzFjlIvshjkvlpopDv_JyVZj8MMCChGpV_jfnL&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2S7ef9Mg$>
> )
>
> at org.apache.tomcat.util.net.SocketProcessorBase.run
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2forg.apache.tomcat.util.net.SocketProcessorBase.run&c=E,1,9zXaF9vHvKVlE5NpCYjiHmbZrmqbG0B_GHQq4rSrzpeeXFCwFUs3MBzaE0j74-TGhcnMER1vqZG2FZFG0laisCtMlBX1R9nEaId_KoNNZqaYDU99jw,,&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt37ct6vGA$>
> (SocketProcessorBase.java:52
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fSocketProcessorBase.java*3a52&c=E,1,LZDkH_krRzHgwiDBEmnF4kCwabJiZvRp890kF2HGgehkBrUkK0u9dIj5At5lRRQzte2_ZWRYj5BFyVm-MOHGtHsq6PLN6c4vGPVzxDxCNw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt39S_Es0w$>
> )
>
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1191
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fThreadPoolExecutor.java*3a1191&c=E,1,sFhkQGGY9T9drnsrqqTCqhYgt4P6sXplDPrKuhStXuq91fnWPdOAqtYJjsHH6ElilKWv63vhvYti2xx4zSQ6lUgtYswUBxUXuOC_L_roshPdeua_KBE,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0UC_pVvQ$>
> )
>
> at org.apache.tomcat.util.threads.ThreadPoolExecutor$
> Worker.run
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fWorker.run&c=E,1,AAJck1rx7huIcGTcZLVifmrqKKqn9tJ8_EiWLY_LgGGq07_vbhVv_TGyx999HyAkXdEQyZ2inpt_xkHU46VsugUd4KNB_lCB-NsAEnOJ9Q,,&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0OfUx4Kw$>
> (ThreadPoolExecutor.java:659
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fThreadPoolExecutor.java*3a659&c=E,1,HYWoCOj08fcNggtMqwtg8befDJ3OvA2zAifg0xQdLJm5e-OnXnaKI5_SCii3WvU3-9HCXwaChGG1bTuavL0vL7KEXHx7D7Xe9-vkD7UfIw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1hFmOFBA$>
> )
>
> at org.apache.tomcat.util.threads.TaskThread$
> WrappingRunnable.run
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fWrappingRunnable.run&c=E,1,-v5NUrxqHwzGg-9dAW4Q5AoU0tfAeo30H8Ky80--szahAOLdpHSZRDXUFiK7lQ3Bf9c5gNtj9N1EPXf5bIt1IkQnBtFxU8AfBBf11UhBIsEZ6D10CQsc3DRj&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3COcN9jw$>
> (TaskThread.java:61
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fTaskThread.java*3a61&c=E,1,uRfcR0WsV0fQqEaa3HYCcLRIMMemHgiaz-A36uGG4a4HY2hQpshoi77gZxVZSANCQkQR10KWG0F8foV7t4Ae3MPXGUCZBhnuwUFqJQ0SvS6KEw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0OboHauQ$>
> )
>
> at java.base/java.lang.Thread.run
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fjava.lang.Thread.run&c=E,1,6mclfOVKxSEC_VTgnVOWXrNwCcwYCqWYCtOBwdKeBT-A6z_62O805-qznvQOW6gHIZG-OKE3p1uXcX6TXJihVOENpw0vbkAd0DvI1_9GU_E,&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0_EVu8fg$>
> (Thread.java:833
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fThread.java*3a833&c=E,1,Xjdr2WgvR9se4-q8Y6ReY1YymidfVI1RqBfB-swMy0sAbyu3LQ3Gj7Oy8oDVv_49jgELRy8Q7kE1ZDlAmd26Yxr2iOdrJ6pPVb2qKYi2NQLH&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0gwDS_4w$>
> )
>
> Caused by:
> ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT
> message=AMQ219014: Timed out after waiting 10000 ms for response when
> sending packet 71]
>
> ... 65 more
>
>
>
>
>
> This is the stack trace at the second timeout message
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> ActiveMQConnection.java:714
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fActiveMQConnection.java*3a714&c=E,1,UTaiF_0hpU4wTQqz4_3iStrmfLszIHizjPpueBR5yv-5ilk0tbhwZ6KgeYpKKCaTEGJZLlNJziUbmKJSjQlA6nTOxihhnoWGSNbToAQ6zOSqEw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1KndddZQ$>
> )
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> ActiveMQConnection.java:735
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fActiveMQConnection.java*3a735&c=E,1,M-b7JHMmR30vR5cCmk3NcNzfkC3m4MBhI7tJ_VStAzCaYBWqPVD1pyvEx9LD-I1LVDZjbHVeBfUMf5HOCEhANjQfJ84JNVn9pXV2_zh-kizyhMD05Yo,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0X4203Dw$>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> ClientSessionFactoryImpl.java:868
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fClientSessionFactoryImpl.java*3a868&c=E,1,htiR1-PgM2OgWpPbflNwb3TOVGuHjNJ46r1zIi_tR55EfzAJFETCEUZrDBv8nvJKD7vDL5puGHA7emmSLPMO_-0M1bkJoUpDZE6Y12NlqAbLdD30blmH1A,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt06tX_5VA$>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> ClientSessionFactoryImpl.java:856
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fClientSessionFactoryImpl.java*3a856&c=E,1,rG1HeinOiFfS-vAJeiDrP_tJ46W3ZwKfb5vP3JU86p789aoYHLT9_cltu1F556YuVfjAXcVJSdMX0vLZwS5Am1c9OjgzGUgiix9431pSXnGAFHg,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3yM95SKg$>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:802
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fClientSessionFactoryImpl.java*3a802&c=E,1,IpmoqWzjRsBWsWFRVyophC7xvobUYm_M-R07j2weNl0fMuXLc1F29q9TuV8AjEY2mxpBZJG12kK5SBibaGPeCrwCi0b0aUt56930OdwTuciH_pWkhR9SLw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3q-oPNhg$>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fClientSessionFactoryImpl.java*3a566&c=E,1,jMFCSfeRebikH-AioEUI7cUUntw6-yuiUplnWJKFhMj_w-XlKte9e8HGTG5EbkHvQOJb6f__nDkxR20RBBxNTJx3swi3wt0dTLBbQ9Gc7YyL36YKbxoXhw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2RA_btVg$>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fClientSessionFactoryImpl.java*3a1407&c=E,1,SgODhYYd-iTgFlhmTmEi87435ahyCVF06rZr29wB4ZmpZT3fVrzB9y4Ukxyt3Qp4spxbR0RlCgSK6IE6eJ9EXZ3psySz49YvNJ1DF88roNqzWGPo8yzx&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0JG-v9YQ$>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fAbstractRemotingConnection.java*3a98&c=E,1,WDcthbTMBqHSmI-aOCsqLih30H8yDiQr_PVd_7nskCH4_kGGOYaGF9nA4AG-8L2m_jDAlXCYT933o1FU8neMfkT9g3kehZBPKrJ8ZpXgURBaYUqcOFCnWMpEQA,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1-IzgpLQ$>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0pYKjuOHMOTL3UaLqDP6EqlqIjMTA8Xs2Y5pF6MJBTUbxDdQxIfbPyCavvMtGGypNVBbRbQ8-zZIrx7GQgfC3Q44sIhPeSqAk5BKHMNnPvxngsHc9VM,&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1ykQsa-Q$>
> (RemotingConnectionImpl.java:212
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fRemotingConnectionImpl.java*3a212&c=E,1,6CSm8GgMGGprYxpYP6IyjwroEMKSTd3ydQtGEw04aqh6Vdzvg4I112zsVdAt5yslOpmIMPxULflUZ7OL0Kim3JxwroK-Mk2vgWjNssTKwQ8,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2WbUq6qw$>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fCloseRunnable.run&c=E,1,eZvshsv6mememDd-0R4ghtzmYVOLSwz0A4FoHwzjGidDTToNBDzZezVfUnRD6-Fr6ZpJ7xgwcwgEUTXDxNytrkZYn7xVe_CCXLYp_k_endKN4c003YNrwdo,&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt1rQHaGMg$>
> (ClientSessionFactoryImpl.java:1172
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fClientSessionFactoryImpl.java*3a1172&c=E,1,E-urG5zq6gIV_DxAli9FjMYUOObRAtZL6qOaFUOOlJR2DFhySGzl0Ezt0enYkygsres1LWIiBlimOgEWcRQLo36pWLNBBOeWfcicG6Mj6XPUQg,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt075-XzRg$>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fOrderedExecutor.java*3a57&c=E,1,9bznxhGY4EfYRHk3swYAryZ9Zv7cr16Ixkl_UjLyd_Rl0kFg3SXDBp0wvdSLtJCAdA5Cz3TwzASQ6lUgVrWM9Wtzwyg98MkPPCXX0xokHw,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3j1smHOQ$>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fOrderedExecutor.java*3a32&c=E,1,t1lXLP_aqiYrMVYyb-bMDpM4mLF7TLWLbeerRhHX63W6TLf1jYUGt25oITkxOoxm4ZoQ3I2GZfQAcFmpZaR3AoFfQ8GxB-rpqID6_0FeePLZAysHVL5S&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3Gip9gYA$>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fProcessorBase.java*3a68&c=E,1,dqTUe0Hw6248go_69PKbcLIlvBQTpHuV3CcCFsk-FiPCM1O48nXbUV_guNPd_jI_IVZQt9VSlZUqjyeDy_6y-re8MDJqbOzruaHVVHU1-ApA5aWpoTmvgnEJo8A,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt3MXWDHKA$>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fThreadPoolExecutor.java*3a1136&c=E,1,2nYUC5WAfJfnAGhEGid8vIIWld0ejYgY8lIQg25Q7UeeR6HyQEUIMvx5AdKZx7iRMN_Jbcqk6Z1y6DUYTgq1Ibe2ohbtuLOuknrK9Q2N&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2WsExbgQ$>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fWorker.run&c=E,1,yYLSxR4Z6kEEBJHWE_U5Y9Yj1yJ5gCTjWnJG93-QTmhagOXIH88mEsnpEaGf68M9tjwmL6Qs2JkjN8ev6sHlkorNqTfDRuDd19bHUwXbIA,,&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0sCtaj5A$>
> (ThreadPoolExecutor.java:635
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fThreadPoolExecutor.java*3a635&c=E,1,2gx_LRp6l1TeZbPuvvGze5ybAfvTxvVesnh004mjFKDVBi_9Ef01UvfMvzF5H0rBG6jvG9JYYuENOCyvYC1mCYQZcTWUVRQS79tVx2dqhA,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2stqsCHw$>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2f1.run&c=E,1,MYVb_RE5tczV-l7jvqgk1CydEYEAoiVV15yENbdhr6da1wfFTwF2_k6H9SruEgixQxL3QD--ryo5iePNIQ7Ve5LWsaWtoIlJ4w3dFSCQuomtJQ,,&typo=1&ancr_add=1__;JSUl!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt2NMmQF-w$>
> (ActiveMQThreadFactory.java:118
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fActiveMQThreadFactory.java*3a118&c=E,1,vb13BY-ZO2U4XSS8_k_oQ8L62v_m40T1EZTt5i_ls8XG5HX-X1d04kNXs9AhVlI65lduhnB5yAjB4aYNPPsm3Sda_IjNqgkbYkoUblIYcHB6j2kQuQ,,&typo=1&ancr_add=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt0fcfagXg$>
> )
>
> Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED
> message=AMQ219015: The connection was disconnected because of server
> shutdown]
>
> ... 7 more
>
>
>
>
>
>
>
> [image: rg]
> <https://urldefense.com/v3/__https://linkprotect.cudasvc.com/url?a=https*3a*2f*2fwww.redpointglobal.com*2f&c=E,1,-sALdzxL74lBcP9YXBclI84IpjFZ6fxlUvA4KGryH_bBqlIZMivu2xrDDFqAM0Oyqr6m7m1ahJp7KQcXuo4ph4YG2nUn_O66R2CmaR3YseKA9dE3QA,,&typo=1__;JSUlJQ!!Ck4v2Rc!nzN3whXfPpxIweqDhJzenIYN05MWZUfHlopo7TBhpb6m8p-5EP7pvzzQD-SMZfQwgBJ_y4NGWq3NEStRIDMAEkW0wjT3tt23XDGlag$>
>
> *John Lilley *
>
> *Data Management Chief Architect, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
