Hi, I would like know if you have worked in the solution of this theme, related with recovered of queues in service cas multiplier, after restart uima-as broker, specifically temp queue. If you have a snapshot with this. Thanks
2016-09-28 10:34 GMT-04:00, Jaroslaw Cwiklik <uim...@gmail.com>: > Nelson, yes I think there is a problem in the UIMA-AS. First, there is > evidence that a service > is trying to use a stale temp queue: > > tcp://localhost:61616 Reason: javax.jms.InvalidDestinationException: > Cannot use a Temporary destination from another Connection. > > > Also, I think there is another problem related to recovery of temp queues > in the service. > The service cas multiplier uses multiple queues: > 1) input queue - receives GetMeta and Process requests from a client > 2) temp queue - receives FreeCas requests from a client. This is only > created when a service > is a Cas Multiplier. This is used to free CASes in a cas multiplier. > > The input queue listener recovers the connection since it received an input > CAS. I think the > temp queue is not recovered and this leads to repeated retries. > > I will work on fixing this today and let you know when this is ready. I > will also create a new > snapshot for you to try. > > Thanks for finding these problems. > > - jerry > > > On Wed, Sep 28, 2016 at 10:12 AM, nelson rivera <nelsonriver...@gmail.com> > wrote: > >> ok, only show you the result of last test. >> Now in my test with uima-as-2.9.0, after complete restart of the >> Broker UIMA-AS log show >> >> 09:51:01.731 - 19: >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer. >> handleListenerSetupFailure: >> WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered >> Connection to Broker: tcp://localhost:61616 - Ready to Process Again >> 09:51:01.740 - 46: >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer. >> handleListenerSetupFailure: >> WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered >> Connection to Broker: tcp://localhost:61616 - Ready to Process Again >> 09:51:01.747 - 15: >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer. >> handleListenerSetupFailure: >> WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered >> Connection to Broker: tcp://localhost:61616 - Ready to Process Again >> 09:51:01.773 - 14: >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont >> ainer.onException: >> WARNING: Service: Aggregate Cluster Analyzer Runtime Exception >> 09:51:01.773 - 14: >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont >> ainer.onException: >> WARNING: Service: Aggregate Cluster Analyzer Runtime Exception >> 09:51:01.774 - 14: >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont >> ainer.onException: >> WARNING: Jms Listener Failed. Endpoint: XClusterAnalyzerAggregate >> Managed By: tcp://localhost:61616 Reason: javax.jms.JMSException: >> java.io.EOFException >> 09:51:01.774 - 14: >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer. >> handleListenerSetupFailure: >> WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Unable To >> Connect To Broker: tcp://localhost:61616 Retrying Until Successful ... >> 09:51:01.777 - 19: >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont >> ainer.onException: >> WARNING: Service: Aggregate Cluster Analyzer Runtime Exception >> 09:51:01.777 - 19: >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont >> ainer.onException: >> WARNING: Jms Listener Failed. Endpoint: >> ID:nelson-H81-M1-43454-1475070490428-1:1:1 Managed By: >> tcp://localhost:61616 Reason: javax.jms.InvalidDestinationException: >> Cannot use a Temporary destination from another Connection. >> >> this lines are continue to repeat in the log indefinitely. And after >> this have the same problem when send input Cas at the service >> >> 2016-09-28 9:27 GMT-04:00, Jaroslaw Cwiklik <uim...@gmail.com>: >> > Nelson, I will investigate your use case and let you know. >> > >> > -jerry >> > >> > On Wed, Sep 28, 2016 at 9:01 AM, nelson rivera >> > <nelsonriver...@gmail.com >> > >> > wrote: >> > >> >> Thank you for describe how UIMA-AS works, it is very helpful. But my >> >> scenario is not restart the broker while the uima-as service works on >> >> input CAS >> >> >> >> my scenario is >> >> 1- Start Broker and UIMA-AS aggregate service >> >> >> >> 2- Complete Restart of the Broker (without restart the service). >> >> UIMA-AS log show the lines that previously i said you: >> >> "Jms Listener Failed." .... >> >> "The Session is closed." ... >> >> "The Consumer is closed." ... >> >> >> >> I do not see in my uima-as log the lines that you say in your test >> >> after broker is bounced: >> >> Listeners reconnect: >> >> INFO | Successfully refreshed JMS Connection >> >> INFO | Successfully refreshed JMS Connection >> >> In my case (uima-as 2.8.1) seems to be a problem with reestablishing >> >> the connection to the broker after restart. >> >> >> >> 4- Then in this point, after the broker was restarted completely, >> >> uima-as client sends a CAS to UIMA-AS aggregate service. And for this >> >> petition UIMA-AS client receive the parent CAS, but not child CAS. >> >> >> >> In this moment i will trys with uima-as-2.9.0 snapshot >> >> >> >> >> >> 2016-09-27 15:33 GMT-04:00, Jaroslaw Cwiklik <uim...@gmail.com>: >> >> > Nelson, in my scenario I do have a Cas Multiplier which produces a >> >> > child >> >> > CAS for each input CAS. For the purpose of this >> >> > test its not important how many internal delegates there are in a >> >> pipeline. >> >> > The service is configured as a Cas Multiplier >> >> > and it does send child CASes when a broker is running. >> >> > >> >> > The problem is that once the input CAS is received, its reply queue >> >> > is >> >> > bound to a specific temp reply queue. When a broker is >> >> > bounced, that queue is not valid anymore and cant be used to return >> >> > child >> >> > CASes nor the input CAS. To guard against such >> >> > cases you need to use timeouts on the client side. >> >> > >> >> > Like I said in my previous email I dont get how an input CAS can be >> >> > returned back to the client after a broker bounce. This should >> >> > not happen since its temp queue does not exist in a new broker. >> >> > >> >> > I am still looking at this scenario as I don't like what I see in >> >> > the >> >> > service log. I think there might be a bug in the UIMA-AS but I am >> >> > still investigating. Bug or not, what I described above is how >> >> > UIMA-AS >> >> > works. You should not expect any replies from a service >> >> > working on a CAS that was sent through a broker that is bounced >> >> > before >> >> > a >> >> > reply. >> >> > >> >> > -jerry >> >> > >> >> > >> >> > On Tue, Sep 27, 2016 at 3:18 PM, nelson rivera >> >> > <nelsonriver...@gmail.com >> >> > >> >> > wrote: >> >> > >> >> >> My service is a aggregate analysis engine that receives input CAS >> >> >> with the direcction of a directory of documents, the first >> >> >> component >> >> >> is a MultiplerCas that for each document generate a child CAS, each >> >> >> child CAS continues the flow that are other primitive annotators >> >> >> (primitive annotators in c++ that can throw exceptions), then >> >> >> exists >> >> >> other MultiplerCas that acts as Merger, waiting the complete >> >> >> processing of each child CAS and finally generate a child CAS where >> >> >> each child CAS previously processed is a view of CAS, this a child >> CAS >> >> >> that is composed of view continúa el flujo a otro AE, que espera un >> >> >> CAS de esta manera. >> >> >> I will send you my uima-as descriptor, to understand better my >> >> >> service. All delegates are co-located. >> >> >> >> >> >> i will trys with uima-as-2.9.0 snapshot how you say, to see how it >> >> >> behaves >> >> >> >> >> >> 2016-09-27 14:27 GMT-04:00, Jaroslaw Cwiklik <uim...@gmail.com>: >> >> >> > I was able to stage this scenario in which an aggregate service >> >> >> > cas >> >> >> > multiplier receives a single input CAS and >> >> >> > produces a child CAS. >> >> >> > >> >> >> > Scenario: >> >> >> > 1) Service receives input CAS >> >> >> > 2) CasMultiplier generates a child CAS >> >> >> > 3) AE.process() is called (child CAS) and it sleeps for awhile >> >> >> > 4) Broker is shutdown >> >> >> > 5) AE.process() awakes from sleep >> >> >> > 6) Uima-AS tries to deliver child CAS to a remote client >> >> >> > 7) Connection is invalid so the child CAS is dropped >> >> >> > 8) Broker is restarted >> >> >> > 9) Listeners reconnect: >> >> >> > INFO | Successfully refreshed JMS Connection >> >> >> > INFO | Successfully refreshed JMS Connection >> >> >> > 10) Broker log shows: >> >> >> > WARN Service - Async error occurred: >> >> >> > org.apache.activemq.DestinationDoesNotExistException: The >> >> >> > destination >> >> >> > temp-queue://ID:bluejws65-53920-1474997439574-8:1:1 does not >> exist. >> >> >> > WARN Service - Async error occurred: >> >> >> > org.apache.activemq.DestinationDoesNotExistException: The >> >> >> > destination >> >> >> > temp-queue://ID:bluejws65-53920-1474997439574-8:1:1 does not >> exist. >> >> >> > >> >> >> > The service failed to send a reply.Nothing is delivered to a >> >> >> > client >> >> >> > as >> >> >> > expected. This is the confusing part since you see a parent CAS >> >> >> > being >> >> >> sent >> >> >> > to a client. I just cant imagine how this is possible. >> >> >> > >> >> >> > When a service receives an input CAS in a message, it extracts a >> JMS >> >> >> > replyTo object which is a JMS temp queue and stores in an >> >> >> > internal >> >> >> > cache >> >> >> > until it is time to send a reply. Temp queues are transient and >> >> >> > unique >> >> >> each >> >> >> > time one is created. >> >> >> > >> >> >> > If the broker dies and subsequently restarts, the cached replyTo >> >> >> > queue >> >> >> > object is invalid (temp queue no longer exist) which leads to >> >> >> > "Async error occurred: >> >> >> > org.apache.activemq.DestinationDoesNotExistException" as shows >> >> >> > above. >> >> >> > >> >> >> > In the service log I see a bunch of these exceptions: >> >> >> > >> >> >> > 02:05:46.574 - 56: >> >> >> > org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl. >> >> >> openChannel: >> >> >> > WARNING: Service: Test Aggregate TAE Unable to Open Connection To >> >> >> > Broker: >> >> >> > tcp://localhost:61616?wireFormat.maxInactivityDuration=0 - >> Silently >> >> >> > Retrying ... >> >> >> > 02:05:46.574 - 56: >> >> >> > org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl. >> >> >> openChannel: >> >> >> > WARNING: >> >> >> > javax.jms.JMSException: Could not connect to broker URL: >> >> >> > tcp://localhost:61616?wireFormat.maxInactivityDuration=0. Reason: >> >> >> > java.net.ConnectException: Connection refused >> >> >> > >> >> >> > This is *after* the service is restarted so there seems to be a >> >> problem >> >> >> > with reestablishing the connection to a new broker. >> >> >> > I am still investigating that part. I just am confused how you >> >> >> > are >> >> able >> >> >> to >> >> >> > see a reply in the client. >> >> >> > >> >> >> > -Jerry >> >> >> > >> >> >> > >> >> >> > On Tue, Sep 27, 2016 at 1:13 PM, Jaroslaw Cwiklik < >> uim...@gmail.com> >> >> >> wrote: >> >> >> > >> >> >> >> From what I understand there are two issues: >> >> >> >> >> >> >> >> 1) Why service JMS listeners are not reconnecting when a broker >> >> >> >> is >> >> >> >> bounced. This >> >> >> >> should be done by default. The 2.9.0 snapshot may address >> >> >> >> this >> >> >> >> since >> >> >> >> a >> >> >> >> listener >> >> >> >> uses newer AMQ jars (5.14.0) >> >> >> >> >> >> >> >> 2) What happens to a child CAS when a Broker disappears and the >> >> >> >> service >> >> >> >> wants >> >> >> >> to deliver it to a client that is not reachable. I need to >> >> >> >> investigate >> >> >> >> this part. I think >> >> >> >> such CAS is dropped but that should be reflected in a log >> >> >> >> and >> >> also >> >> >> >> the >> >> >> >> input CAS >> >> >> >> should be marked as failed. Again something for me to >> >> investigate. >> >> >> >> >> >> >> >> I will try to set up a test to see if I can create the >> >> >> >> scenario.It >> >> >> >> will >> >> >> >> take me a bit of time >> >> >> >> to set this up but I will let you know what happens. >> >> >> >> >> >> >> >> -jerry >> >> >> >> >> >> >> >> On Tue, Sep 27, 2016 at 12:42 PM, Jaroslaw Cwiklik >> >> >> >> <uim...@gmail.com >> >> > >> >> >> >> wrote: >> >> >> >> >> >> >> >>> Here is the uima-as 2.9.0-SNAPSHOT to try: >> >> >> >>> https://repository.apache.org/content/repositories/snapshots >> >> >> >>> /org/apache/uima/uima-as/2.9.0-SNAPSHOT/uima-as-2.9.0-201609 >> >> >> >>> 23.195233-6.tar.gz >> >> >> >>> >> >> >> >>> Can you briefly describe your service? Does your service have >> >> >> >>> any >> >> >> remote >> >> >> >>> delegates or all parts of the pipeline are in the same process? >> >> >> >>> Do you see and exceptions in the service log? >> >> >> >>> >> >> >> >>> -jerry >> >> >> >>> >> >> >> >>> On Tue, Sep 27, 2016 at 9:07 AM, nelson rivera >> >> >> >>> <nelsonriver...@gmail.com> >> >> >> >>> wrote: >> >> >> >>> >> >> >> >>>> I was testing with AMQ broker 5.14.0 and i get the same error. >> >> >> >>>> not >> >> >> >>>> appears to be broker >> >> >> >>>> issues, appears to be uima-as issues. Any other idea, i am >> >> >> >>>> stagnant >> >> >> >>>> at >> >> >> >>>> work >> >> >> >>>> >> >> >> >>>> 2016-09-22 13:04 GMT-04:00, Jaroslaw Cwiklik <uim...@gmail.com >> >: >> >> >> >>>> > Can you install AMQ broker 5.14.0 to see if this is related >> >> >> >>>> > to >> >> >> broker >> >> >> >>>> > issues. >> >> >> >>>> > >> >> >> >>>> > -jerry >> >> >> >>>> > >> >> >> >>>> > On Thu, Sep 22, 2016 at 11:29 AM, nelson rivera < >> >> >> >>>> nelsonriver...@gmail.com> >> >> >> >>>> > wrote: >> >> >> >>>> > >> >> >> >>>> >> just like that, the parent CAS comes back but the child CAS >> >> >> >>>> >> doesnt. >> >> >> >>>> >> >> >> >> >>>> >> And the child CAS if processed, because the uima-as log >> >> >> >>>> >> shows >> >> >> >>>> >> the >> >> >> >>>> >> lines of the other annotators primitive of the flow >> processing >> >> >> >>>> >> the >> >> >> >>>> >> child CAS, but not comes back to UIMA-AS client. >> >> >> >>>> >> >> >> >> >>>> >> uima-as log only shows >> >> >> >>>> >> "Jms Listener Failed. Endpoint: >> >> >> >>>> >> temp-queue://ID:localhsot-H81-M1-33610-1474481197190-1:1:1 >> >> >> >>>> >> Managed >> >> >> >>>> By: >> >> >> >>>> >> tcp://localhost:61616 Reason: javax.jms.JMSException: >> >> >> >>>> >> java.io.EOFException" ........ >> >> >> >>>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: >> >> >> >>>> >> XClusterAnalyzerAggregate >> >> >> >>>> >> Managed By: tcp://localhost:61616 Reason: >> >> >> >>>> >> javax.jms.IllegalStateException: The Consumer is >> >> >> >>>> >> closed.............. >> >> >> >>>> >> >> >> >> >>>> >> The CAS is send to service after restart the Broker, and >> >> >> >>>> >> how >> >> >> >>>> >> say >> >> >> >>>> >> parent CAS comes back but the child CAS doesnt. And if i >> >> >> >>>> >> try >> >> >> quiesce >> >> >> >>>> >> the Aggregate Service console shows : "Quiescing UIMA-AS >> >> Service. >> >> >> >>>> >> Remaining Number of CASes to Process:1". I suppose that is >> >> >> >>>> >> the >> >> >> >>>> >> child >> >> >> >>>> >> CAS. >> >> >> >>>> >> >> >> >> >>>> >> the client if can send more CASes to the service, but not >> >> receive >> >> >> >>>> >> response, for each CAS send after the one CAS that was send >> >> after >> >> >> >>>> >> restart broker (for which parent CAS comes back but the >> child >> >> >> >>>> >> CAS >> >> >> >>>> >> doesnt) not comes back parent CAS or child CAS. >> >> >> >>>> >> >> >> >> >>>> >> More specific: >> >> >> >>>> >> scenario 1: >> >> >> >>>> >> 1- Start Broker and UIMA-AS aggregate service. Everything >> >> >> >>>> >> is >> >> >> >>>> >> fine. >> >> >> >>>> >> >> >> >> >>>> >> 2- Restart Broker (without restart the service). UIMA-AS >> >> >> >>>> >> log >> >> >> >>>> >> show >> >> >> >>>> >> : >> >> >> >>>> >> "Jms Listener Failed." .... >> >> >> >>>> >> 3- Client sends a CAS to UIMA-AS aggregate service.(The >> parent >> >> >> >>>> >> CAS >> >> >> >>>> >> comes back but the child CAS doesnt, to the uima-as >> >> >> >>>> >> client.) >> >> >> >>>> >> If i try quiesce the Aggregate Service console shows : >> >> "Quiescing >> >> >> >>>> >> UIMA-AS Service. Remaining Number of CASes to Process:1" >> >> >> >>>> >> >> >> >> >>>> >> scenario 2: >> >> >> >>>> >> 1- Start Broker and UIMA-AS aggregate service. Everything >> >> >> >>>> >> is >> >> >> >>>> >> fine. >> >> >> >>>> >> >> >> >> >>>> >> 2- Restart Broker (without restart the service). UIMA-AS >> >> >> >>>> >> log >> >> >> >>>> >> show >> >> >> >>>> >> : >> >> >> >>>> >> "Jms Listener Failed." .... >> >> >> >>>> >> 3- Client sends a CAS to UIMA-AS aggregate service.(The >> parent >> >> >> >>>> >> CAS >> >> >> >>>> >> comes back but the child CAS doesnt, to the uima-as client) >> >> >> >>>> >> 4- Client sends another CAS to UIMA-AS aggregate >> service.(not >> >> >> comes >> >> >> >>>> >> back parent CAS or child CAS , to the uima-as client) >> >> >> >>>> >> 5- Client sends another CAS to UIMA-AS aggregate >> service.(not >> >> >> comes >> >> >> >>>> >> back parent CAS or child CAS , to the uima-as client) >> >> >> >>>> >> If i try quiesce the Aggregate Service console shows: >> >> >> >>>> >> "Quiescing >> >> >> >>>> >> UIMA-AS Service. Remaining Number of CASes to Process:5" >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> 2016-09-22 10:50 GMT-04:00, Jaroslaw Cwiklik < >> uim...@gmail.com >> >> >: >> >> >> >>>> >> > When a parent CAS comes back (after broker is restored), >> can >> >> >> >>>> >> > the >> >> >> >>>> client >> >> >> >>>> >> > send more CASes to the service and receive responses? I >> >> >> >>>> >> > am >> >> >> >>>> >> > trying >> >> >> >>>> >> > to >> >> >> >>>> >> > determine what the real problem is here. >> >> >> >>>> >> > >> >> >> >>>> >> > 1) JMS Listener not recovering connection >> >> >> >>>> >> > 2) UIMA-AS is not handling CAS reply failures when broker >> >> >> >>>> connection is >> >> >> >>>> >> > stale >> >> >> >>>> >> > >> >> >> >>>> >> > -jerry >> >> >> >>>> >> > >> >> >> >>>> >> > On Thu, Sep 22, 2016 at 10:38 AM, Jaroslaw Cwiklik < >> >> >> >>>> uim...@gmail.com> >> >> >> >>>> >> > wrote: >> >> >> >>>> >> > >> >> >> >>>> >> >> Strange. You say that the parent CAS comes back but the >> >> >> >>>> >> >> child >> >> >> CAS >> >> >> >>>> >> doesnt. >> >> >> >>>> >> >> Are there any exceptions in the service log? Perhaps the >> >> >> >>>> >> >> service >> >> >> >>>> >> >> is >> >> >> >>>> >> >> unable >> >> >> >>>> >> >> to send the child CAS due to missing broker. There >> >> >> >>>> >> >> should >> >> >> >>>> >> >> be >> >> >> >>>> something >> >> >> >>>> >> in >> >> >> >>>> >> >> the uima-as log about this. >> >> >> >>>> >> >> >> >> >> >>>> >> >> -jerry >> >> >> >>>> >> >> >> >> >> >>>> >> >> On Thu, Sep 22, 2016 at 10:19 AM, nelson rivera >> >> >> >>>> >> >> <nelsonriver...@gmail.com> >> >> >> >>>> >> >> wrote: >> >> >> >>>> >> >> >> >> >> >>>> >> >>> Yes, it is supposed to do the service, given a >> >> >> >>>> >> >>> CAS it produces a child CAS and and this CAS continue >> >> >> >>>> >> >>> in >> >> >> >>>> >> >>> the >> >> >> >>>> >> >>> flow, >> >> >> >>>> >> >>> that are other primitive annotators. And I expected >> >> >> >>>> >> >>> that >> >> >> >>>> >> >>> the >> >> >> >>>> >> >>> child >> >> >> >>>> >> >>> CAS >> >> >> >>>> >> >>> will sent to UIMA-AS client, but not do. >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> the scenario: >> >> >> >>>> >> >>> 1- Start Broker and UIMA-AS aggregate service >> >> >> >>>> >> >>> 2- Client sends a CAS to UIMA-AS aggregate service to >> >> >> >>>> >> >>> verify >> >> >> >>>> >> >>> everything is fine. The process is correct. >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> 3- Restart Broker (without restart the service). >> >> >> >>>> >> >>> UIMA-AS >> >> log >> >> >> >>>> show : >> >> >> >>>> >> >>> "Jms Listener Failed." .... >> >> >> >>>> >> >>> 4- Client sends a CAS to UIMA-AS aggregate service >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> UIMA-AS client receive the parent CAS, but not child >> >> >> >>>> >> >>> CAS. >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> Thank for you quick reply >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> 2016-09-22 9:48 GMT-04:00, Jaroslaw Cwiklik < >> >> uim...@gmail.com >> >> >> >: >> >> >> >>>> >> >>> > Can you clarify if the UIMA-AS service is a Cas >> >> Multiplier. >> >> >> >>>> Meaning >> >> >> >>>> >> >>> given a >> >> >> >>>> >> >>> > CAS it produces child CASes >> >> >> >>>> >> >>> > which are then sent to UIMA-AS client. Is this what >> this >> >> >> >>>> service is >> >> >> >>>> >> >>> > supposed to do if the broker is running? >> >> >> >>>> >> >>> > >> >> >> >>>> >> >>> > Can you confirm the scenario: >> >> >> >>>> >> >>> > 1) Client sends a CAS to UIMA-AS aggregate service >> >> >> >>>> >> >>> > 2) Service receives the CAS >> >> >> >>>> >> >>> > 3) Broker dies >> >> >> >>>> >> >>> > 4) Broker is restarted >> >> >> >>>> >> >>> > >> >> >> >>>> >> >>> > Does the UIMA-AS client receive the CAS back in a >> >> >> >>>> >> >>> > reply >> >> >> >>>> >> >>> > when >> >> >> a >> >> >> >>>> >> >>> > broker >> >> >> >>>> >> >>> > is >> >> >> >>>> >> >>> > restarted? >> >> >> >>>> >> >>> > >> >> >> >>>> >> >>> > -Jerry >> >> >> >>>> >> >>> > >> >> >> >>>> >> >>> > >> >> >> >>>> >> >>> > >> >> >> >>>> >> >>> > >> >> >> >>>> >> >>> > On Thu, Sep 22, 2016 at 9:33 AM, nelson rivera < >> >> >> >>>> >> >>> nelsonriver...@gmail.com> >> >> >> >>>> >> >>> > wrote: >> >> >> >>>> >> >>> > >> >> >> >>>> >> >>> >> I will try the variants that you say. I was >> >> >> >>>> >> >>> >> deepening >> >> >> >>>> >> >>> >> the >> >> >> >>>> problem: >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> I have a Aggregate AE that contains CasMultiplier, >> >> >> >>>> >> >>> >> the >> >> >> >>>> >> >>> >> original >> >> >> >>>> >> >>> >> CAS >> >> >> >>>> >> >>> >> is finish when it arrives to this CasMultiplier, >> >> >> >>>> >> >>> >> this >> >> >> >>>> generates >> >> >> >>>> >> >>> >> new >> >> >> >>>> >> >>> >> CAS and is this CAS that continue in the flow of >> >> >> >>>> >> >>> >> Aggregate. >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> After restart broker (without restart the service of >> >> >> >>>> >> >>> >> annotator) >> >> >> >>>> >> >>> >> and >> >> >> >>>> >> >>> >> send a CAS to process, the listener associated to >> >> >> >>>> >> >>> >> the >> >> >> client >> >> >> >>>> api >> >> >> >>>> >> >>> >> uima-as (BaseUIMAAsynchronousEngine_impl) is >> >> >> >>>> >> >>> >> notified >> >> >> >>>> >> >>> >> when >> >> >> >>>> the >> >> >> >>>> >> >>> >> original CAS finalizes, but the new Cas generated in >> >> >> >>>> >> >>> >> the >> >> >> >>>> >> >>> >> CasMultiplier, It is not notifies when finalizes. >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> with Aggregate AE simple the log is the same, but >> >> >> >>>> >> >>> >> how >> >> >> >>>> >> >>> >> the >> >> >> >>>> original >> >> >> >>>> >> >>> >> CAS >> >> >> >>>> >> >>> >> is notifies into the listener when finalizes, not >> have >> >> >> >>>> >> >>> >> problem >> >> >> >>>> >> >>> >> because is not expected for another new CAS. >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> 2016-09-21 17:23 GMT-04:00, Jaroslaw Cwiklik >> >> >> >>>> >> >>> >> <uim...@gmail.com >> >> >> >>>> >: >> >> >> >>>> >> >>> >> > If possible can you download AMQ 5.14.0 (latest) >> >> >> >>>> >> >>> >> > and >> >> use >> >> >> >>>> >> >>> >> > that >> >> >> >>>> >> >>> >> > broker >> >> >> >>>> >> >>> >> > instead of the one the uima-as 2.8.1 ships with? >> >> >> >>>> >> >>> >> > There >> >> >> >>>> >> >>> >> > was >> >> >> >>>> >> >>> >> > a >> >> >> >>>> >> >>> >> > nasty >> >> >> >>>> >> >>> bug >> >> >> >>>> >> >>> >> > in >> >> >> >>>> >> >>> >> > AMQ related to keep-alive protocol not working. I >> >> >> >>>> >> >>> >> > am >> >> not >> >> >> >>>> sure if >> >> >> >>>> >> >>> >> > this >> >> >> >>>> >> >>> >> > is >> >> >> >>>> >> >>> >> > the cause but worth trying. >> >> >> >>>> >> >>> >> > >> >> >> >>>> >> >>> >> > You can download AMQ from here: >> >> >> >>>> >> >>> >> > http://activemq.apache.org/download.html >> >> >> >>>> >> >>> >> > >> >> >> >>>> >> >>> >> > -Jerry >> >> >> >>>> >> >>> >> > >> >> >> >>>> >> >>> >> > On Wed, Sep 21, 2016 at 4:52 PM, Jaroslaw Cwiklik >> >> >> >>>> >> >>> >> > <uim...@gmail.com> >> >> >> >>>> >> >>> >> wrote: >> >> >> >>>> >> >>> >> > >> >> >> >>>> >> >>> >> >> Not sure what is causing this. I need to stage >> >> >> >>>> >> >>> >> >> this >> >> >> >>>> >> >>> >> >> with >> >> >> >>>> >> >>> >> >> UIMA-AS >> >> >> >>>> >> >>> >> >> 2.8.1. >> >> >> >>>> >> >>> >> I >> >> >> >>>> >> >>> >> >> am currently testing with trunk (latest) code. >> >> >> >>>> >> >>> >> >> The >> >> most >> >> >> >>>> current >> >> >> >>>> >> >>> >> >> code >> >> >> >>>> >> >>> >> does >> >> >> >>>> >> >>> >> >> have newer broker. >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> I've started a broker and a sample service: >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> $UIMA_HOME/bin/deployAsyncService.sh >> >> >> >>>> >> >>> >> >> $UIMA_HOME/examples/deploy/as/ >> >> >> >>>> >> >>> >> >> Deploy_MeetingDetectorTAE.xml >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> After the service started, I stopped the broker, >> >> waited >> >> >> >>>> >> >>> >> >> a >> >> >> >>>> bit, >> >> >> >>>> >> and >> >> >> >>>> >> >>> >> >> restarted the broker. >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> With UIMA-AS from trunk I see the following in >> >> >> >>>> >> >>> >> >> the >> >> log: >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> 04:21:35.496 - 1: org.apache.uima.adapter.jms. >> >> activemq. >> >> >> >>>> >> >>> >> >> SpringContainerDeployer.doStartListeners: INFO: >> >> >> >>>> >> >>> >> >> Controller: >> >> >> >>>> >> >>> >> >> Meeting >> >> >> >>>> >> >>> >> >> Detector TAE Trying to Start Listener on >> >> >> >>>> >> >>> >> >> Endpoint: >> >> >> >>>> >> >>> >> >> queue://MeetingDetectorTaeQueue >> >> >> >>>> >> >>> >> >> Selector: Command=2000 OR Command=2002 Broker: >> >> >> >>>> >> >>> >> >> tcp://localhost:61616 >> >> >> >>>> >> >>> >> >> 04:21:35.873 - 1: org.apache.uima.adapter.jms. >> >> activemq. >> >> >> >>>> >> >>> >> >> SpringContainerDeployer.doStartListeners: INFO: >> >> >> >>>> >> >>> >> >> Controller: >> >> >> >>>> >> >>> >> >> Meeting >> >> >> >>>> >> >>> >> >> Detector TAE Trying to Start Listener on >> >> >> >>>> >> >>> >> >> Endpoint: >> >> >> >>>> >> >>> >> >> queue://MeetingDetectorTaeQueue >> >> >> >>>> >> >>> >> >> Selector: Command=2001 Broker: >> tcp://localhost:61616 >> >> >> >>>> >> >>> >> >> 04:21:45.621 - 22: >> >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. >> >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: >> >> >> WARNING: >> >> >> >>>> >> Service: >> >> >> >>>> >> >>> >> >> Meeting Detector TAE Runtime Exception >> >> >> >>>> >> >>> >> >> 04:21:45.622 - 22: >> >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. >> >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: >> >> >> WARNING: >> >> >> >>>> Jms >> >> >> >>>> >> >>> Listener >> >> >> >>>> >> >>> >> >> Failed. Endpoint: MeetingDetectorTaeQueue Managed >> >> >> >>>> >> >>> >> >> By: >> >> >> >>>> >> >>> >> >> tcp://localhost:61616 >> >> >> >>>> >> >>> >> >> Reason: javax.jms.JMSException: >> java.io.EOFException >> >> >> >>>> >> >>> >> >> 04:21:45.622 - 22: >> >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. >> >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerCont >> >> >> >>>> ainer.handleListenerSetupFailure: >> >> >> >>>> >> >>> >> WARNING: >> >> >> >>>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener >> Unable >> >> To >> >> >> >>>> Connect >> >> >> >>>> >> To >> >> >> >>>> >> >>> >> >> Broker: >> >> >> >>>> >> >>> >> >> tcp://localhost:61616 Retrying Until Successful >> >> >> >>>> >> >>> >> >> ... >> >> >> >>>> >> >>> >> >> 04:21:45.638 - 27: >> >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. >> >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: >> >> >> WARNING: >> >> >> >>>> >> Service: >> >> >> >>>> >> >>> >> >> Meeting Detector TAE Runtime Exception >> >> >> >>>> >> >>> >> >> 04:21:45.639 - 27: >> >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. >> >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: >> >> >> WARNING: >> >> >> >>>> Jms >> >> >> >>>> >> >>> Listener >> >> >> >>>> >> >>> >> >> Failed. Endpoint: MeetingDetectorTaeQueue Managed >> >> >> >>>> >> >>> >> >> By: >> >> >> >>>> >> >>> >> >> tcp://localhost:61616 >> >> >> >>>> >> >>> >> >> Reason: javax.jms.JMSException: >> java.io.EOFException >> >> >> >>>> >> >>> >> >> 04:21:45.639 - 27: >> >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. >> >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerCont >> >> >> >>>> ainer.handleListenerSetupFailure: >> >> >> >>>> >> >>> >> WARNING: >> >> >> >>>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener >> Unable >> >> To >> >> >> >>>> Connect >> >> >> >>>> >> To >> >> >> >>>> >> >>> >> >> Broker: >> >> >> >>>> >> >>> >> >> tcp://localhost:61616 Retrying Until Successful >> >> >> >>>> >> >>> >> >> ... >> >> >> >>>> >> >>> >> >> 04:22:07.439 - 22: >> >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. >> >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerCont >> >> >> >>>> ainer.handleListenerSetupFailure: >> >> >> >>>> >> >>> >> WARNING: >> >> >> >>>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener >> >> Recovered >> >> >> >>>> >> Connection >> >> >> >>>> >> >>> to >> >> >> >>>> >> >>> >> >> Broker: tcp://localhost:61616 - Ready to Process >> >> >> >>>> >> >>> >> >> Again >> >> >> >>>> >> >>> >> >> 04:22:07.454 - 27: >> >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. >> >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerCont >> >> >> >>>> ainer.handleListenerSetupFailure: >> >> >> >>>> >> >>> >> WARNING: >> >> >> >>>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener >> >> Recovered >> >> >> >>>> >> Connection >> >> >> >>>> >> >>> to >> >> >> >>>> >> >>> >> >> Broker: tcp://localhost:61616 - Ready to Process >> >> >> >>>> >> >>> >> >> Again >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> I will repeat the above test with v 2.8.1 >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> -Jerry >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> On Wed, Sep 21, 2016 at 4:34 PM, nelson rivera < >> >> >> >>>> >> >>> >> nelsonriver...@gmail.com> >> >> >> >>>> >> >>> >> >> wrote: >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >>> I'm using UIMA-AS 2.8.1, any idea that what >> happen? >> >> >> >>>> >> >>> >> >>> >> >> >> >>>> >> >>> >> >>> 2016-09-21 16:09 GMT-04:00, Jaroslaw Cwiklik < >> >> >> >>>> uim...@gmail.com >> >> >> >>>> >> >: >> >> >> >>>> >> >>> >> >>> > Which version of UIMA-AS are you using? >> >> >> >>>> >> >>> >> >>> > >> >> >> >>>> >> >>> >> >>> > -Jerry >> >> >> >>>> >> >>> >> >>> > >> >> >> >>>> >> >>> >> >>> > On Wed, Sep 21, 2016 at 3:42 PM, nelson rivera >> >> >> >>>> >> >>> >> >>> > < >> >> >> >>>> >> >>> >> >>> nelsonriver...@gmail.com> >> >> >> >>>> >> >>> >> >>> > wrote: >> >> >> >>>> >> >>> >> >>> > >> >> >> >>>> >> >>> >> >>> >> When start ActiveMQ and deploys an annotator >> >> >> >>>> >> >>> >> >>> >> as >> >> >> >>>> >> >>> >> >>> >> service, >> >> >> >>>> >> >>> >> >>> >> processing >> >> >> >>>> >> >>> >> >>> >> is >> >> >> >>>> >> >>> >> >>> >> executed correctly, but when the broker is >> >> stopped >> >> >> >>>> >> >>> >> >>> >> and >> >> >> >>>> >> >>> >> >>> >> subsequently >> >> >> >>>> >> >>> >> >>> >> restarted. UIMA-AS log show the lines below >> and >> >> >> after >> >> >> >>>> this >> >> >> >>>> >> >>> >> >>> >> the >> >> >> >>>> >> >>> >> >>> >> api >> >> >> >>>> >> >>> >> >>> >> client uima-as is not notified in the >> >> >> >>>> >> >>> >> >>> >> listener >> >> when >> >> >> >>>> >> >>> >> >>> >> a >> >> >> >>>> cas >> >> >> >>>> >> >>> >> >>> >> is >> >> >> >>>> >> >>> >> >>> >> processed, any more. >> >> >> >>>> >> >>> >> >>> >> can help me?: >> >> >> >>>> >> >>> >> >>> >> >> >> >> >>>> >> >>> >> >>> >> 02:08:25.570 - 14: >> >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac >> >> >> >>>> tivemq.UimaDefaultMessageListe >> >> >> >>>> >> >>> nerCont >> >> >> >>>> >> >>> >> >>> >> ainer.onException: >> >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster >> Analyzer >> >> >> >>>> >> >>> >> >>> >> Runtime >> >> >> >>>> >> >>> Exception >> >> >> >>>> >> >>> >> >>> >> 02:08:25.570 - 14: >> >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac >> >> >> >>>> tivemq.UimaDefaultMessageListe >> >> >> >>>> >> >>> nerCont >> >> >> >>>> >> >>> >> >>> >> ainer.onException: >> >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: >> >> >> >>>> >> >>> >> XClusterAnalyzerAggregate >> >> >> >>>> >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason: >> >> >> >>>> >> >>> javax.jms.JMSException: >> >> >> >>>> >> >>> >> >>> >> java.io.EOFException >> >> >> >>>> >> >>> >> >>> >> 02:08:25.570 - 14: >> >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac >> >> >> >>>> tivemq.UimaDefaultMessageListe >> >> >> >>>> >> >>> >> >>> nerContainer. >> >> >> >>>> >> >>> >> >>> >> handleListenerSetupFailure: >> >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Uima AS Service:Aggregate >> >> >> >>>> >> >>> >> >>> >> Cluster >> >> >> >>>> >> >>> >> >>> >> Analyzer >> >> >> >>>> >> >>> >> >>> >> Listener >> >> >> >>>> >> >>> >> >>> >> Unable To Connect To Broker: >> >> >> >>>> >> >>> >> >>> >> tcp://localhost:61616 >> >> >> >>>> Retrying >> >> >> >>>> >> >>> >> >>> >> ... >> >> >> >>>> >> >>> >> >>> >> QueueFailure: ADVERTENCIA: Jms Listener >> >> >> >>>> >> >>> >> >>> >> Failed. >> >> >> >>>> Endpoint: >> >> >> >>>> >> >>> >> >>> >> temp-queue://ID:localhost-H81- >> >> >> >>>> M1-33610-1474481197190-1:1:1 >> >> >> >>>> >> >>> Managed >> >> >> >>>> >> >>> >> >>> >> By: >> >> >> >>>> >> >>> >> >>> >> tcp://localhost:61616 Reason: >> >> >> >>>> >> >>> >> >>> >> javax.jms.JMSException: >> >> >> >>>> >> >>> >> >>> >> java.io.EOFException >> >> >> >>>> >> >>> >> >>> >> 02:09:02.222 - 14: >> >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac >> >> >> >>>> tivemq.UimaDefaultMessageListe >> >> >> >>>> >> >>> >> >>> nerContainer. >> >> >> >>>> >> >>> >> >>> >> handleListenerSetupFailure: >> >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Uima AS Service:Aggregate >> >> >> >>>> >> >>> >> >>> >> Cluster >> >> >> >>>> >> >>> >> >>> >> Analyzer >> >> >> >>>> >> >>> >> >>> >> Listener >> >> >> >>>> >> >>> >> >>> >> Established Connection to Broker: >> >> >> >>>> >> >>> >> >>> >> tcp://localhost:61616 >> >> >> >>>> >> >>> >> >>> >> >> >> >> >>>> >> >>> >> >>> >> 02:09:02.317 - 14: >> >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac >> >> >> >>>> tivemq.UimaDefaultMessageListe >> >> >> >>>> >> >>> nerCont >> >> >> >>>> >> >>> >> >>> >> ainer.onException: >> >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster >> Analyzer >> >> >> >>>> >> >>> >> >>> >> Runtime >> >> >> >>>> >> >>> Exception >> >> >> >>>> >> >>> >> >>> >> 02:09:02.317 - 14: >> >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac >> >> >> >>>> tivemq.UimaDefaultMessageListe >> >> >> >>>> >> >>> nerCont >> >> >> >>>> >> >>> >> >>> >> ainer.onException: >> >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: >> >> >> >>>> >> >>> >> XClusterAnalyzerAggregate >> >> >> >>>> >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason: >> >> >> >>>> >> >>> >> >>> >> org.apache.activemq.ConnectionClosedException: >> >> The >> >> >> >>>> >> connection >> >> >> >>>> >> >>> is >> >> >> >>>> >> >>> >> >>> >> already closed02:09:02.356 - 15: >> >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac >> >> >> >>>> tivemq.UimaDefaultMessageListe >> >> >> >>>> >> >>> nerCont >> >> >> >>>> >> >>> >> >>> >> ainer.onException: >> >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster >> Analyzer >> >> >> >>>> >> >>> >> >>> >> Runtime >> >> >> >>>> >> >>> Exception >> >> >> >>>> >> >>> >> >>> >> 02:09:02.357 - 15: >> >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac >> >> >> >>>> tivemq.UimaDefaultMessageListe >> >> >> >>>> >> >>> nerCont >> >> >> >>>> >> >>> >> >>> >> ainer.onException: >> >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: >> >> >> >>>> >> >>> >> XClusterAnalyzerAggregate >> >> >> >>>> >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason: >> >> >> >>>> >> >>> >> >>> >> javax.jms.IllegalStateException: The Consumer >> is >> >> >> >>>> closed. >> >> >> >>>> >> >>> >> >>> >> >> >> >> >>>> >> >>> >> >>> > >> >> >> >>>> >> >>> >> >>> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> > >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> > >> >> >> >>>> >> >>> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> > >> >> >> >>>> >> >> >> >> >>>> > >> >> >> >>>> >> >> >> >>> >> >> >> >>> >> >> >> >> >> >> >> > >> >> >> >> >> > >> >> >> > >> >
