>From what I understand there are two issues:
1) Why service JMS listeners are not reconnecting when a broker is bounced.
This
should be done by default. The 2.9.0 snapshot may address this since a
listener
uses newer AMQ jars (5.14.0)
2) What happens to a child CAS when a Broker disappears and the service
wants
to deliver it to a client that is not reachable. I need to investigate
this part. I think
such CAS is dropped but that should be reflected in a log and also the
input CAS
should be marked as failed. Again something for me to investigate.
I will try to set up a test to see if I can create the scenario.It will
take me a bit of time
to set this up but I will let you know what happens.
-jerry
On Tue, Sep 27, 2016 at 12:42 PM, Jaroslaw Cwiklik <[email protected]> wrote:
> Here is the uima-as 2.9.0-SNAPSHOT to try:
> https://repository.apache.org/content/repositories/snapshots
> /org/apache/uima/uima-as/2.9.0-SNAPSHOT/uima-as-2.9.0-
> 20160923.195233-6.tar.gz
>
> Can you briefly describe your service? Does your service have any remote
> delegates or all parts of the pipeline are in the same process?
> Do you see and exceptions in the service log?
>
> -jerry
>
> On Tue, Sep 27, 2016 at 9:07 AM, nelson rivera <[email protected]>
> wrote:
>
>> I was testing with AMQ broker 5.14.0 and i get the same error. not
>> appears to be broker
>> issues, appears to be uima-as issues. Any other idea, i am stagnant at
>> work
>>
>> 2016-09-22 13:04 GMT-04:00, Jaroslaw Cwiklik <[email protected]>:
>> > Can you install AMQ broker 5.14.0 to see if this is related to broker
>> > issues.
>> >
>> > -jerry
>> >
>> > On Thu, Sep 22, 2016 at 11:29 AM, nelson rivera <
>> [email protected]>
>> > wrote:
>> >
>> >> just like that, the parent CAS comes back but the child CAS doesnt.
>> >>
>> >> And the child CAS if processed, because the uima-as log shows the
>> >> lines of the other annotators primitive of the flow processing the
>> >> child CAS, but not comes back to UIMA-AS client.
>> >>
>> >> uima-as log only shows
>> >> "Jms Listener Failed. Endpoint:
>> >> temp-queue://ID:localhsot-H81-M1-33610-1474481197190-1:1:1 Managed By:
>> >> tcp://localhost:61616 Reason: javax.jms.JMSException:
>> >> java.io.EOFException" ........
>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: XClusterAnalyzerAggregate
>> >> Managed By: tcp://localhost:61616 Reason:
>> >> javax.jms.IllegalStateException: The Consumer is closed..............
>> >>
>> >> The CAS is send to service after restart the Broker, and how say
>> >> parent CAS comes back but the child CAS doesnt. And if i try quiesce
>> >> the Aggregate Service console shows : "Quiescing UIMA-AS Service.
>> >> Remaining Number of CASes to Process:1". I suppose that is the child
>> >> CAS.
>> >>
>> >> the client if can send more CASes to the service, but not receive
>> >> response, for each CAS send after the one CAS that was send after
>> >> restart broker (for which parent CAS comes back but the child CAS
>> >> doesnt) not comes back parent CAS or child CAS.
>> >>
>> >> More specific:
>> >> scenario 1:
>> >> 1- Start Broker and UIMA-AS aggregate service. Everything is fine.
>> >>
>> >> 2- Restart Broker (without restart the service). UIMA-AS log show :
>> >> "Jms Listener Failed." ....
>> >> 3- Client sends a CAS to UIMA-AS aggregate service.(The parent CAS
>> >> comes back but the child CAS doesnt, to the uima-as client.)
>> >> If i try quiesce the Aggregate Service console shows : "Quiescing
>> >> UIMA-AS Service. Remaining Number of CASes to Process:1"
>> >>
>> >> scenario 2:
>> >> 1- Start Broker and UIMA-AS aggregate service. Everything is fine.
>> >>
>> >> 2- Restart Broker (without restart the service). UIMA-AS log show :
>> >> "Jms Listener Failed." ....
>> >> 3- Client sends a CAS to UIMA-AS aggregate service.(The parent CAS
>> >> comes back but the child CAS doesnt, to the uima-as client)
>> >> 4- Client sends another CAS to UIMA-AS aggregate service.(not comes
>> >> back parent CAS or child CAS , to the uima-as client)
>> >> 5- Client sends another CAS to UIMA-AS aggregate service.(not comes
>> >> back parent CAS or child CAS , to the uima-as client)
>> >> If i try quiesce the Aggregate Service console shows: "Quiescing
>> >> UIMA-AS Service. Remaining Number of CASes to Process:5"
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> 2016-09-22 10:50 GMT-04:00, Jaroslaw Cwiklik <[email protected]>:
>> >> > When a parent CAS comes back (after broker is restored), can the
>> client
>> >> > send more CASes to the service and receive responses? I am trying to
>> >> > determine what the real problem is here.
>> >> >
>> >> > 1) JMS Listener not recovering connection
>> >> > 2) UIMA-AS is not handling CAS reply failures when broker connection
>> is
>> >> > stale
>> >> >
>> >> > -jerry
>> >> >
>> >> > On Thu, Sep 22, 2016 at 10:38 AM, Jaroslaw Cwiklik <[email protected]
>> >
>> >> > wrote:
>> >> >
>> >> >> Strange. You say that the parent CAS comes back but the child CAS
>> >> doesnt.
>> >> >> Are there any exceptions in the service log? Perhaps the service is
>> >> >> unable
>> >> >> to send the child CAS due to missing broker. There should be
>> something
>> >> in
>> >> >> the uima-as log about this.
>> >> >>
>> >> >> -jerry
>> >> >>
>> >> >> On Thu, Sep 22, 2016 at 10:19 AM, nelson rivera
>> >> >> <[email protected]>
>> >> >> wrote:
>> >> >>
>> >> >>> Yes, it is supposed to do the service, given a
>> >> >>> CAS it produces a child CAS and and this CAS continue in the flow,
>> >> >>> that are other primitive annotators. And I expected that the child
>> >> >>> CAS
>> >> >>> will sent to UIMA-AS client, but not do.
>> >> >>>
>> >> >>> the scenario:
>> >> >>> 1- Start Broker and UIMA-AS aggregate service
>> >> >>> 2- Client sends a CAS to UIMA-AS aggregate service to verify
>> >> >>> everything is fine. The process is correct.
>> >> >>>
>> >> >>> 3- Restart Broker (without restart the service). UIMA-AS log
>> show :
>> >> >>> "Jms Listener Failed." ....
>> >> >>> 4- Client sends a CAS to UIMA-AS aggregate service
>> >> >>>
>> >> >>> UIMA-AS client receive the parent CAS, but not child CAS.
>> >> >>>
>> >> >>> Thank for you quick reply
>> >> >>>
>> >> >>> 2016-09-22 9:48 GMT-04:00, Jaroslaw Cwiklik <[email protected]>:
>> >> >>> > Can you clarify if the UIMA-AS service is a Cas Multiplier.
>> Meaning
>> >> >>> given a
>> >> >>> > CAS it produces child CASes
>> >> >>> > which are then sent to UIMA-AS client. Is this what this service
>> is
>> >> >>> > supposed to do if the broker is running?
>> >> >>> >
>> >> >>> > Can you confirm the scenario:
>> >> >>> > 1) Client sends a CAS to UIMA-AS aggregate service
>> >> >>> > 2) Service receives the CAS
>> >> >>> > 3) Broker dies
>> >> >>> > 4) Broker is restarted
>> >> >>> >
>> >> >>> > Does the UIMA-AS client receive the CAS back in a reply when a
>> >> >>> > broker
>> >> >>> > is
>> >> >>> > restarted?
>> >> >>> >
>> >> >>> > -Jerry
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > On Thu, Sep 22, 2016 at 9:33 AM, nelson rivera <
>> >> >>> [email protected]>
>> >> >>> > wrote:
>> >> >>> >
>> >> >>> >> I will try the variants that you say. I was deepening the
>> problem:
>> >> >>> >>
>> >> >>> >> I have a Aggregate AE that contains CasMultiplier, the original
>> >> >>> >> CAS
>> >> >>> >> is finish when it arrives to this CasMultiplier, this generates
>> >> >>> >> new
>> >> >>> >> CAS and is this CAS that continue in the flow of Aggregate.
>> >> >>> >>
>> >> >>> >> After restart broker (without restart the service of annotator)
>> >> >>> >> and
>> >> >>> >> send a CAS to process, the listener associated to the client
>> api
>> >> >>> >> uima-as (BaseUIMAAsynchronousEngine_impl) is notified when the
>> >> >>> >> original CAS finalizes, but the new Cas generated in the
>> >> >>> >> CasMultiplier, It is not notifies when finalizes.
>> >> >>> >>
>> >> >>> >> with Aggregate AE simple the log is the same, but how the
>> original
>> >> >>> >> CAS
>> >> >>> >> is notifies into the listener when finalizes, not have problem
>> >> >>> >> because is not expected for another new CAS.
>> >> >>> >>
>> >> >>> >> 2016-09-21 17:23 GMT-04:00, Jaroslaw Cwiklik <[email protected]
>> >:
>> >> >>> >> > If possible can you download AMQ 5.14.0 (latest) and use that
>> >> >>> >> > broker
>> >> >>> >> > instead of the one the uima-as 2.8.1 ships with? There was a
>> >> >>> >> > nasty
>> >> >>> bug
>> >> >>> >> > in
>> >> >>> >> > AMQ related to keep-alive protocol not working. I am not sure
>> if
>> >> >>> >> > this
>> >> >>> >> > is
>> >> >>> >> > the cause but worth trying.
>> >> >>> >> >
>> >> >>> >> > You can download AMQ from here:
>> >> >>> >> > http://activemq.apache.org/download.html
>> >> >>> >> >
>> >> >>> >> > -Jerry
>> >> >>> >> >
>> >> >>> >> > On Wed, Sep 21, 2016 at 4:52 PM, Jaroslaw Cwiklik
>> >> >>> >> > <[email protected]>
>> >> >>> >> wrote:
>> >> >>> >> >
>> >> >>> >> >> Not sure what is causing this. I need to stage this with
>> >> >>> >> >> UIMA-AS
>> >> >>> >> >> 2.8.1.
>> >> >>> >> I
>> >> >>> >> >> am currently testing with trunk (latest) code. The most
>> current
>> >> >>> >> >> code
>> >> >>> >> does
>> >> >>> >> >> have newer broker.
>> >> >>> >> >>
>> >> >>> >> >> I've started a broker and a sample service:
>> >> >>> >> >>
>> >> >>> >> >> $UIMA_HOME/bin/deployAsyncService.sh
>> >> >>> >> >> $UIMA_HOME/examples/deploy/as/
>> >> >>> >> >> Deploy_MeetingDetectorTAE.xml
>> >> >>> >> >>
>> >> >>> >> >> After the service started, I stopped the broker, waited a
>> bit,
>> >> and
>> >> >>> >> >> restarted the broker.
>> >> >>> >> >>
>> >> >>> >> >> With UIMA-AS from trunk I see the following in the log:
>> >> >>> >> >>
>> >> >>> >> >> 04:21:35.496 - 1: org.apache.uima.adapter.jms.activemq.
>> >> >>> >> >> SpringContainerDeployer.doStartListeners: INFO: Controller:
>> >> >>> >> >> Meeting
>> >> >>> >> >> Detector TAE Trying to Start Listener on Endpoint:
>> >> >>> >> >> queue://MeetingDetectorTaeQueue
>> >> >>> >> >> Selector: Command=2000 OR Command=2002 Broker:
>> >> >>> >> >> tcp://localhost:61616
>> >> >>> >> >> 04:21:35.873 - 1: org.apache.uima.adapter.jms.activemq.
>> >> >>> >> >> SpringContainerDeployer.doStartListeners: INFO: Controller:
>> >> >>> >> >> Meeting
>> >> >>> >> >> Detector TAE Trying to Start Listener on Endpoint:
>> >> >>> >> >> queue://MeetingDetectorTaeQueue
>> >> >>> >> >> Selector: Command=2001 Broker: tcp://localhost:61616
>> >> >>> >> >> 04:21:45.621 - 22: org.apache.uima.adapter.jms.activemq.
>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: WARNING:
>> >> Service:
>> >> >>> >> >> Meeting Detector TAE Runtime Exception
>> >> >>> >> >> 04:21:45.622 - 22: org.apache.uima.adapter.jms.activemq.
>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: WARNING:
>> Jms
>> >> >>> Listener
>> >> >>> >> >> Failed. Endpoint: MeetingDetectorTaeQueue Managed By:
>> >> >>> >> >> tcp://localhost:61616
>> >> >>> >> >> Reason: javax.jms.JMSException: java.io.EOFException
>> >> >>> >> >> 04:21:45.622 - 22: org.apache.uima.adapter.jms.activemq.
>> >> >>> >> >> UimaDefaultMessageListenerContainer.handleListenerSetupFailu
>> re:
>> >> >>> >> WARNING:
>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener Unable To
>> Connect
>> >> To
>> >> >>> >> >> Broker:
>> >> >>> >> >> tcp://localhost:61616 Retrying Until Successful ...
>> >> >>> >> >> 04:21:45.638 - 27: org.apache.uima.adapter.jms.activemq.
>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: WARNING:
>> >> Service:
>> >> >>> >> >> Meeting Detector TAE Runtime Exception
>> >> >>> >> >> 04:21:45.639 - 27: org.apache.uima.adapter.jms.activemq.
>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: WARNING:
>> Jms
>> >> >>> Listener
>> >> >>> >> >> Failed. Endpoint: MeetingDetectorTaeQueue Managed By:
>> >> >>> >> >> tcp://localhost:61616
>> >> >>> >> >> Reason: javax.jms.JMSException: java.io.EOFException
>> >> >>> >> >> 04:21:45.639 - 27: org.apache.uima.adapter.jms.activemq.
>> >> >>> >> >> UimaDefaultMessageListenerContainer.handleListenerSetupFailu
>> re:
>> >> >>> >> WARNING:
>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener Unable To
>> Connect
>> >> To
>> >> >>> >> >> Broker:
>> >> >>> >> >> tcp://localhost:61616 Retrying Until Successful ...
>> >> >>> >> >> 04:22:07.439 - 22: org.apache.uima.adapter.jms.activemq.
>> >> >>> >> >> UimaDefaultMessageListenerContainer.handleListenerSetupFailu
>> re:
>> >> >>> >> WARNING:
>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener Recovered
>> >> Connection
>> >> >>> to
>> >> >>> >> >> Broker: tcp://localhost:61616 - Ready to Process Again
>> >> >>> >> >> 04:22:07.454 - 27: org.apache.uima.adapter.jms.activemq.
>> >> >>> >> >> UimaDefaultMessageListenerContainer.handleListenerSetupFailu
>> re:
>> >> >>> >> WARNING:
>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener Recovered
>> >> Connection
>> >> >>> to
>> >> >>> >> >> Broker: tcp://localhost:61616 - Ready to Process Again
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >> I will repeat the above test with v 2.8.1
>> >> >>> >> >>
>> >> >>> >> >> -Jerry
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >> On Wed, Sep 21, 2016 at 4:34 PM, nelson rivera <
>> >> >>> >> [email protected]>
>> >> >>> >> >> wrote:
>> >> >>> >> >>
>> >> >>> >> >>> I'm using UIMA-AS 2.8.1, any idea that what happen?
>> >> >>> >> >>>
>> >> >>> >> >>> 2016-09-21 16:09 GMT-04:00, Jaroslaw Cwiklik <
>> [email protected]
>> >> >:
>> >> >>> >> >>> > Which version of UIMA-AS are you using?
>> >> >>> >> >>> >
>> >> >>> >> >>> > -Jerry
>> >> >>> >> >>> >
>> >> >>> >> >>> > On Wed, Sep 21, 2016 at 3:42 PM, nelson rivera <
>> >> >>> >> >>> [email protected]>
>> >> >>> >> >>> > wrote:
>> >> >>> >> >>> >
>> >> >>> >> >>> >> When start ActiveMQ and deploys an annotator as service,
>> >> >>> >> >>> >> processing
>> >> >>> >> >>> >> is
>> >> >>> >> >>> >> executed correctly, but when the broker is stopped and
>> >> >>> >> >>> >> subsequently
>> >> >>> >> >>> >> restarted. UIMA-AS log show the lines below and after
>> this
>> >> >>> >> >>> >> the
>> >> >>> >> >>> >> api
>> >> >>> >> >>> >> client uima-as is not notified in the listener when a cas
>> >> >>> >> >>> >> is
>> >> >>> >> >>> >> processed, any more.
>> >> >>> >> >>> >> can help me?:
>> >> >>> >> >>> >>
>> >> >>> >> >>> >> 02:08:25.570 - 14:
>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac
>> tivemq.UimaDefaultMessageListe
>> >> >>> nerCont
>> >> >>> >> >>> >> ainer.onException:
>> >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster Analyzer Runtime
>> >> >>> Exception
>> >> >>> >> >>> >> 02:08:25.570 - 14:
>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac
>> tivemq.UimaDefaultMessageListe
>> >> >>> nerCont
>> >> >>> >> >>> >> ainer.onException:
>> >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint:
>> >> >>> >> XClusterAnalyzerAggregate
>> >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason:
>> >> >>> javax.jms.JMSException:
>> >> >>> >> >>> >> java.io.EOFException
>> >> >>> >> >>> >> 02:08:25.570 - 14:
>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac
>> tivemq.UimaDefaultMessageListe
>> >> >>> >> >>> nerContainer.
>> >> >>> >> >>> >> handleListenerSetupFailure:
>> >> >>> >> >>> >> ADVERTENCIA: Uima AS Service:Aggregate Cluster Analyzer
>> >> >>> >> >>> >> Listener
>> >> >>> >> >>> >> Unable To Connect To Broker: tcp://localhost:61616
>> Retrying
>> >> >>> >> >>> >> ...
>> >> >>> >> >>> >> QueueFailure: ADVERTENCIA: Jms Listener Failed. Endpoint:
>> >> >>> >> >>> >> temp-queue://ID:localhost-H81-
>> M1-33610-1474481197190-1:1:1
>> >> >>> Managed
>> >> >>> >> >>> >> By:
>> >> >>> >> >>> >> tcp://localhost:61616 Reason: javax.jms.JMSException:
>> >> >>> >> >>> >> java.io.EOFException
>> >> >>> >> >>> >> 02:09:02.222 - 14:
>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac
>> tivemq.UimaDefaultMessageListe
>> >> >>> >> >>> nerContainer.
>> >> >>> >> >>> >> handleListenerSetupFailure:
>> >> >>> >> >>> >> ADVERTENCIA: Uima AS Service:Aggregate Cluster Analyzer
>> >> >>> >> >>> >> Listener
>> >> >>> >> >>> >> Established Connection to Broker: tcp://localhost:61616
>> >> >>> >> >>> >>
>> >> >>> >> >>> >> 02:09:02.317 - 14:
>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac
>> tivemq.UimaDefaultMessageListe
>> >> >>> nerCont
>> >> >>> >> >>> >> ainer.onException:
>> >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster Analyzer Runtime
>> >> >>> Exception
>> >> >>> >> >>> >> 02:09:02.317 - 14:
>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac
>> tivemq.UimaDefaultMessageListe
>> >> >>> nerCont
>> >> >>> >> >>> >> ainer.onException:
>> >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint:
>> >> >>> >> XClusterAnalyzerAggregate
>> >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason:
>> >> >>> >> >>> >> org.apache.activemq.ConnectionClosedException: The
>> >> connection
>> >> >>> is
>> >> >>> >> >>> >> already closed02:09:02.356 - 15:
>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac
>> tivemq.UimaDefaultMessageListe
>> >> >>> nerCont
>> >> >>> >> >>> >> ainer.onException:
>> >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster Analyzer Runtime
>> >> >>> Exception
>> >> >>> >> >>> >> 02:09:02.357 - 15:
>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac
>> tivemq.UimaDefaultMessageListe
>> >> >>> nerCont
>> >> >>> >> >>> >> ainer.onException:
>> >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint:
>> >> >>> >> XClusterAnalyzerAggregate
>> >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason:
>> >> >>> >> >>> >> javax.jms.IllegalStateException: The Consumer is closed.
>> >> >>> >> >>> >>
>> >> >>> >> >>> >
>> >> >>> >> >>>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >
>> >> >>> >>
>> >> >>> >
>> >> >>>
>> >> >>
>> >> >>
>> >> >
>> >>
>> >
>>
>
>
