Hi, can you include part of a client logging just before the broker is stopped and restarted until timeouts start. When a connection is recovered, the client creates a new temp queue and includes it in an outgoing messages to services so that replies can come back. Seems like the client outgoing messages contain a reference to an old temp queue which explain the timeouts. You say you have multiple clients in your app and only one instance works after the broker is restarted. The rest of the clients continuously fail after the broker restart (true?). All UIMA-AS clients share the same broker connection so the question is why only one of the clients recovered and included correct temp queue for replies and the others did not. Can yo confirm that you keep sending messages from your clients and only one of them succeeds with send and receive.
Jerry On Tue, Jul 18, 2017 at 4:09 PM, nelson rivera <[email protected]> wrote: > i just tested the snapshot build from > https://svn.apache.org/repos/asf/uima/uima-as/trunk, i continue using > 8 service uima-as remote, 2 are used only for > aggregates services, and created 6 uima-as client instances in my > client application general. > > Now the behavior is the follow: > > After restart the broker, the first uima-as client that is consumed > (no matter which), > has a correct behavior, the instruccion to uima-as client > "BaseUIMAAsynchronousEngine_impl.connectionOpen()" return "false" and > correctly initializes the client again and process the request. > > All next uima-as client to consume returns "true" to the instruccion > "BaseUIMAAsynchronousEngine_impl.connectionOpen()", send the request, > and then the uima-as service process the request but after shows lines > similar to this in the log > > 03:24:52.338 - 1283: > org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl. > UimaAsAsyncCallbackListener.onException(): > INFO: Service:XCharsetDetector JMS unable to Deliver > CAS:46623412:15d57234659:-7fea - > Error:temp-queue://ID:nelson-XPS-8700-40232-1500405516005-1:1:5 > 03:24:52.338 - 1283: > org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl. > UimaAsAsyncCallbackListener.onException(): > INFO: Controller: XCharsetDetector Handling Release CAS Request. > Releasing CAS Reference Id: 46623412:15d57234659:-7fea > > > and then in my mechanism de timeout in client side, after 10 seconds > without get any notification on onBeforeProcessCAS() metod of uima-as > client's UimaAsBaseCallbackListener y throws timeout exception. > > > > > > 2017-07-17 13:37 GMT-04:00, Jaroslaw Cwiklik <[email protected]>: > > Yes. the changes are in the svn so you prefer building from source just > do > > svn update to pickup the latest code. > > > > Jerry > > > > On Mon, Jul 17, 2017 at 1:18 PM, nelson rivera <[email protected] > > > > wrote: > > > >> Sorry for the delay, I took holidays days, i will test the snapshot very > >> now. > >> > >> I can build the snapshot from > >> https://svn.apache.org/repos/asf/uima/uima-as/trunk?, the change is > >> there? > >> > >> 2017-07-14 9:28 GMT-04:00, Jaroslaw Cwiklik <[email protected]>: > >> > Hi, please download a new snapshot > >> > > >> > https://repository.apache.org/content/repositories/ > >> snapshots/org/apache/uima/uima-as/2.9.1-SNAPSHOT/uima- > >> as-2.9.1-20170714.131835-3.tgz > >> > or > >> > https://repository.apache.org/content/repositories/ > >> snapshots/org/apache/uima/uima-as/2.9.1-SNAPSHOT/uima- > >> as-2.9.1-20170714.131915-4.zip > >> > > >> > You can check signatures of the above here: > >> > https://repository.apache.org/content/repositories/ > >> snapshots/org/apache/uima/uima-as/2.9.1-SNAPSHOT/ > >> > > >> > uima-as-2.9.1-20170714.131835-3.tgz.md5 > >> > uima-as-2.9.1-20170714.131835-3.tgz.sha1 > >> > > >> > uima-as-2.9.1-20170714.131915-4.zip.md5 > >> > uima-as-2.9.1-20170714.131915-4.zip.sha1 > >> > > >> > > >> > > >> > On Tue, Jul 11, 2017 at 10:17 AM, nelson rivera < > >> [email protected]> > >> > wrote: > >> > > >> >> Thanks very much, actually i have my application in production but > >> >> with this bug as reported. i wait > >> >> > >> >> 2017-07-11 9:59 GMT-04:00, Jaroslaw Cwiklik <[email protected]>: > >> >> > Hi, I am investigating this issue now. Are you having this problem > >> >> > in > >> >> > production or testing? The multi-client recovery is quite > >> >> > complicated > >> >> > and > >> >> > subject to race conditions. I need to first re-create the problem > >> >> > and > >> >> work > >> >> > on a solution if this in fact is a bug. From what you sent it looks > >> >> > like > >> >> > there is a bug in client recovery. This may take some time to sort > >> out. > >> >> > Will let you know what I find. > >> >> > > >> >> > Jerry > >> >> > > >> >> > On Tue, Jul 11, 2017 at 9:02 AM, nelson rivera > >> >> > <[email protected] > >> >> > > >> >> > wrote: > >> >> > > >> >> >> analyzing more deep the issue, I have observed that after restart > >> >> >> the > >> >> >> broker, the first uima-as client that is consumed (no matter > >> >> >> which), > >> >> >> has a correct behavior, the instruccion to uima-as client > >> >> >> "BaseUIMAAsynchronousEngine_impl.connectionOpen()" return "false" > >> and > >> >> >> correctly initializes the client again and process the request. > >> >> >> > >> >> >> All next uima-as client to consume returns "true" to the > >> >> >> instruccion > >> >> >> "BaseUIMAAsynchronousEngine_impl.connectionOpen()" and them get > the > >> >> >> error: > >> >> >> > >> >> >> jul 11, 2017 8:11:52 AM > >> >> >> org.apache.uima.adapter.jms.client.ActiveMQMessageSender > >> createSession > >> >> >> INFORMACIÓN: UIMA AS Client Failed Creating Session To > Destination: > >> >> >> XCharsetDetector Managed By Broker: tcp://nelson-XPS-8700:61616 > >> >> >> jul 11, 2017 8:11:52 AM > >> >> >> org.apache.uima.adapter.jms.client.ActiveMQMessageSender > >> createSession > >> >> >> INFORMACIÓN: UIMA AS Client Connection To Destination: > >> >> >> XCharsetDetector Managed By Broker: tcp://nelson-XPS-8700:61616 Is > >> >> >> Closed > >> >> >> jul 11, 2017 8:11:52 AM > >> >> >> org.apache.uima.adapter.jms.client.ActiveMQMessageSender run > >> >> >> ADVERTENCIA: > >> >> >> org.apache.activemq.ConnectionClosedException: The connection is > >> >> already > >> >> >> closed > >> >> >> at org.apache.activemq.ActiveMQConnection.checkClosed( > >> >> >> ActiveMQConnection.java:1459) > >> >> >> at org.apache.activemq.ActiveMQConnection. > >> checkClosedOrFailed( > >> >> >> ActiveMQConnection.java:1446) > >> >> >> at org.apache.activemq.ActiveMQConnection.createSession( > >> >> >> ActiveMQConnection.java:328) > >> >> >> at > >> >> >> org.apache.uima.adapter.jms.client.ActiveMQMessageSender. > >> >> >> createSession(ActiveMQMessageSender.java:108) > >> >> >> at > >> >> >> org.apache.uima.adapter.jms.client.ActiveMQMessageSender. > >> >> >> initializeProducer(ActiveMQMessageSender.java:143) > >> >> >> at > >> >> >> org.apache.uima.adapter.jms.client.ActiveMQMessageSender. > >> >> >> createTextMessage(ActiveMQMessageSender.java:191) > >> >> >> at > >> >> >> org.apache.uima.adapter.jms.client.BaseMessageSender.run( > >> >> >> BaseMessageSender.java:330) > >> >> >> at java.lang.Thread.run(Thread.java:748) > >> >> >> > >> >> >> 2017-07-11 8:50 GMT-04:00, nelson rivera > >> >> >> <[email protected]>: > >> >> >> > My client application not receive any exception in any time. I > >> >> >> > use > >> >> >> > both, in some uima-as client use sendAndReceive() and in others > >> use > >> >> >> > send(), each uima-as client has try-catch and none of them > >> >> >> > receive > >> a > >> >> >> > exception, the following lines are printed in the console > >> output,but > >> >> >> > the uima-as client is not notified and keeps waiting. > >> >> >> > > >> >> >> > 2017-07-10 16:48 GMT-04:00, Jaroslaw Cwiklik <[email protected] > >: > >> >> >> >> it looks like the UIMA-AS client is setup to throw an exception > >> >> >> >> back > >> >> >> >> to > >> >> >> >> the > >> >> >> >> application in this case. Did your application receive an > >> >> >> >> exception? > >> >> >> >> Are > >> >> >> >> you using sendAndReceive() or send()? The UIMA-AS client > rejects > >> >> >> requests > >> >> >> >> while a connection is down. > >> >> >> >> > >> >> >> >> Jerry > >> >> >> >> > >> >> >> >> On Mon, Jul 10, 2017 at 4:19 PM, Jaroslaw Cwiklik < > >> >> [email protected]> > >> >> >> >> wrote: > >> >> >> >> > >> >> >> >>> Hmm, this is a different exception from Session closed. This > >> >> >> >>> time > >> >> >> >>> it > >> >> >> >>> is > >> >> >> >>> a > >> >> >> >>> Connection is closed. Seems like a race condition. How many > >> >> >> >>> UIMA-AS > >> >> >> >>> clients > >> >> >> >>> do you have running in this process? > >> >> >> >>> > >> >> >> >>> Jerry > >> >> >> >>> > >> >> >> >>> On Mon, Jul 10, 2017 at 3:51 PM, nelson rivera > >> >> >> >>> <[email protected]> > >> >> >> >>> wrote: > >> >> >> >>> > >> >> >> >>>> i just tested the snapshot build from > >> >> >> >>>> https://svn.apache.org/repos/asf/uima/uima-as/trunk, and the > >> >> >> >>>> behavior > >> >> >> >>>> still are similar for many uima-as client, end in a > exception, > >> >> >> >>>> but > >> >> >> >>>> now > >> >> >> >>>> show this stacktrace: > >> >> >> >>>> > >> >> >> >>>> org.apache.activemq.ConnectionClosedException: The > connection > >> is > >> >> >> >>>> already > >> >> >> >>>> closed > >> >> >> >>>> at org.apache.activemq.ActiveMQConnection. > >> >> >> checkClosed(ActiveMQC > >> >> >> >>>> onnection.java:1459) > >> >> >> >>>> at org.apache.activemq.ActiveMQConnection. > >> >> >> checkClosedOrFailed(A > >> >> >> >>>> ctiveMQConnection.java:1446) > >> >> >> >>>> at org.apache.activemq.ActiveMQConnection. > >> >> >> createSession(ActiveM > >> >> >> >>>> QConnection.java:328) > >> >> >> >>>> at org.apache.uima.adapter.jms. > >> >> client.ActiveMQMessageSender. > >> >> >> cre > >> >> >> >>>> ateSession(ActiveMQMessageSender.java:108) > >> >> >> >>>> at org.apache.uima.adapter.jms. > >> >> client.ActiveMQMessageSender. > >> >> >> ini > >> >> >> >>>> tializeProducer(ActiveMQMessageSender.java:143) > >> >> >> >>>> at org.apache.uima.adapter.jms. > >> >> client.ActiveMQMessageSender. > >> >> >> cre > >> >> >> >>>> ateTextMessage(ActiveMQMessageSender.java:191) > >> >> >> >>>> at org.apache.uima.adapter.jms. > >> >> client.BaseMessageSender.run( > >> >> >> Bas > >> >> >> >>>> eMessageSender.java:330) > >> >> >> >>>> at java.lang.Thread.run(Thread.java:748) > >> >> >> >>>> > >> >> >> >>>> 2017-07-07 15:33 GMT-04:00, nelson rivera > >> >> >> >>>> <[email protected]>: > >> >> >> >>>> > Sorry for the delay, i am building the framework from the > >> >> >> >>>> > source > >> >> >> >>>> > at > >> >> >> >>>> > https://svn.apache.org/repos/asf/uima/uima-as/trunk, when > >> test > >> >> the > >> >> >> >>>> > issue, i will comment the result. > >> >> >> >>>> > > >> >> >> >>>> > 2017-07-06 10:04 GMT-04:00, Jaroslaw Cwiklik > >> >> >> >>>> > <[email protected] > >> >> >: > >> >> >> >>>> >> Hello, the snapshot binaries (zip + tar) are here: > >> >> >> >>>> >> > >> >> >> >>>> >> https://repository.apache.org/ > content/repositories/snapshots > >> >> >> >>>> /org/apache/uima/uima-as/2.9.1-SNAPSHOT/ > >> >> >> >>>> >> > >> >> >> >>>> >> Jerry > >> >> >> >>>> >> > >> >> >> >>>> >> On Thu, Jul 6, 2017 at 8:51 AM, Jaroslaw Cwiklik > >> >> >> >>>> >> <[email protected]> > >> >> >> >>>> >> wrote: > >> >> >> >>>> >> > >> >> >> >>>> >>> Hi, instructions how to build from source are here: > >> >> >> >>>> >>> https://uima.apache.org/building-uima.html > >> >> >> >>>> >>> > >> >> >> >>>> >>> It's a bit involved process so I will try to post a > >> >> >> >>>> >>> snapshot > >> >> >> >>>> >>> of > >> >> >> >>>> uima-as > >> >> >> >>>> >>> to > >> >> >> >>>> >>> Apache snapshot repository and will provide you with a > >> >> >> >>>> >>> link > >> >> >> >>>> >>> later > >> >> >> >>>> today. > >> >> >> >>>> >>> > >> >> >> >>>> >>> Jerry > >> >> >> >>>> >>> > >> >> >> >>>> >>> On Wed, Jul 5, 2017 at 4:32 PM, nelson rivera < > >> >> >> >>>> [email protected]> > >> >> >> >>>> >>> wrote: > >> >> >> >>>> >>> > >> >> >> >>>> >>>> Please can you give me the url to source code with the > >> >> changes, > >> >> >> to > >> >> >> >>>> >>>> test. > >> >> >> >>>> >>>> > >> >> >> >>>> >>>> 2017-07-05 16:19 GMT-04:00, Jaroslaw Cwiklik > >> >> >> >>>> >>>> <[email protected] > >> >> >> >: > >> >> >> >>>> >>>> > Just delivered a few changes to deal with a stale > >> Session. > >> >> >> >>>> >>>> > Can > >> >> >> >>>> >>>> > you > >> >> >> >>>> >>>> > build > >> >> >> >>>> >>>> > from source and test. > >> >> >> >>>> >>>> > Jerry > >> >> >> >>>> >>>> > > >> >> >> >>>> >>>> > On Wed, Jul 5, 2017 at 2:21 PM, Jaroslaw Cwiklik < > >> >> >> >>>> [email protected]> > >> >> >> >>>> >>>> > wrote: > >> >> >> >>>> >>>> > > >> >> >> >>>> >>>> >> Thanks. Will check the code and work on a fix > >> >> >> >>>> >>>> >> > >> >> >> >>>> >>>> >> Jerry > >> >> >> >>>> >>>> >> > >> >> >> >>>> >>>> >> On Wed, Jul 5, 2017 at 2:10 PM, nelson rivera < > >> >> >> >>>> >>>> [email protected]> > >> >> >> >>>> >>>> >> wrote: > >> >> >> >>>> >>>> >> > >> >> >> >>>> >>>> >>> Here the stack trace for the error that asked me: > >> >> >> >>>> >>>> >>> jul 05, 2017 2:08:50 PM > >> >> >> >>>> >>>> >>> org.apache.uima.adapter.jms. > >> client.ActiveMQMessageSender > >> >> >> >>>> >>>> >>> run > >> >> >> >>>> >>>> >>> ADVERTENCIA: > >> >> >> >>>> >>>> >>> javax.jms.IllegalStateException: The Session is > >> >> >> >>>> >>>> >>> closed > >> >> >> >>>> >>>> >>> at org.apache.activemq.ActiveMQSe > >> >> >> >>>> >>>> ssion.checkClosed(ActiveMQSess > >> >> >> >>>> >>>> >>> ion.java:771) > >> >> >> >>>> >>>> >>> at org.apache.activemq.ActiveMQSe > >> >> >> >>>> >>>> ssion.configureMessage(ActiveM > >> >> >> >>>> >>>> >>> QSession.java:759) > >> >> >> >>>> >>>> >>> at org.apache.activemq.ActiveMQSe > >> >> >> >>>> >>>> ssion.createTextMessage(Active > >> >> >> >>>> >>>> >>> MQSession.java:440) > >> >> >> >>>> >>>> >>> at org.apache.uima.adapter.jms.cl > >> >> >> >>>> >>>> ient.ActiveMQMessageSender.cre > >> >> >> >>>> >>>> >>> ateTextMessage(ActiveMQMessageSender.java:184) > >> >> >> >>>> >>>> >>> at org.apache.uima.adapter.jms.cl > >> >> >> >>>> >>>> ient.BaseMessageSender.run(Bas > >> >> >> >>>> >>>> >>> eMessageSender.java:307) > >> >> >> >>>> >>>> >>> at java.lang.Thread.run(Thread.java:748) > >> >> >> >>>> >>>> >>> > >> >> >> >>>> >>>> >>> 2017-07-05 12:16 GMT-04:00, nelson rivera > >> >> >> >>>> >>>> >>> <[email protected]>: > >> >> >> >>>> >>>> >>> > This behavior is random, not occurs always. In the > >> >> >> >>>> >>>> >>> > most > >> >> >> >>>> >>>> >>> > of > >> >> >> >>>> >>>> >>> > the > >> >> >> >>>> >>>> simple > >> >> >> >>>> >>>> >>> > programs and simple test case tested, after > restart > >> >> >> >>>> >>>> >>> > the > >> >> >> >>>> >>>> >>> > UIMA-AS > >> >> >> >>>> >>>> Broker > >> >> >> >>>> >>>> >>> > without restart service uima-as, the instruccion > to > >> >> >> >>>> >>>> >>> > uima-as > >> >> >> >>>> >>>> >>> > client > >> >> >> >>>> >>>> >>> > "BaseUIMAAsynchronousEngine_ > impl.connectionOpen()" > >> >> return > >> >> >> >>>> "false" > >> >> >> >>>> >>>> and > >> >> >> >>>> >>>> >>> > correctly i initialize the client again. > >> >> >> >>>> >>>> >>> > > >> >> >> >>>> >>>> >>> > The incorrect behavior is most frequently with > many > >> >> >> >>>> >>>> >>> > uima-as > >> >> >> >>>> >>>> >>> > client > >> >> >> >>>> >>>> >>> > instances. Is the case of my general application > in > >> >> which > >> >> >> >>>> >>>> >>> > i > >> >> >> >>>> have > >> >> >> >>>> >>>> many > >> >> >> >>>> >>>> >>> > service uima-as, and for each one creates one > >> >> >> >>>> >>>> >>> > instace > >> >> >> >>>> >>>> >>> > of > >> >> >> >>>> uima-as > >> >> >> >>>> >>>> >>> > client (BaseUIMAAsynchronousEngine_impl) and keep > >> >> >> >>>> >>>> >>> > it > >> >> >> >>>> >>>> >>> > alive > >> >> >> >>>> >>>> >>> > to > >> >> >> >>>> >>>> process > >> >> >> >>>> >>>> >>> > all requests of the service uima-as. > >> >> >> >>>> >>>> >>> > > >> >> >> >>>> >>>> >>> > Is in this scenario with many instancies of > uima-as > >> >> >> >>>> >>>> >>> > client > >> >> >> >>>> >>>> >>> > (BaseUIMAAsynchronousEngine_impl) in the same > >> program > >> >> (a > >> >> >> >>>> >>>> >>> > monolithic > >> >> >> >>>> >>>> >>> > application web in tomcat) where after restart the > >> >> >> >>>> >>>> >>> > UIMA-AS > >> >> >> >>>> Broker > >> >> >> >>>> >>>> >>> > without restart any service uima-as, many uima-as > >> >> >> >>>> >>>> >>> > client > >> >> >> >>>> returns > >> >> >> >>>> >>>> >>> > "true" to the instruccion > >> >> >> >>>> >>>> >>> > "BaseUIMAAsynchronousEngine_ > impl.connectionOpen()" > >> but > >> >> >> them > >> >> >> >>>> throw > >> >> >> >>>> >>>> >>> > "javax.jms.IllegalStateException: The Session is > >> >> closed" > >> >> >> >>>> >>>> >>> > when proces the request. > >> >> >> >>>> >>>> >>> > > >> >> >> >>>> >>>> >>> > 2017-07-04 16:13 GMT-04:00, nelson rivera > >> >> >> >>>> >>>> >>> > <[email protected] > >> >> >> >>>> >>>> >: > >> >> >> >>>> >>>> >>> >> One peculiarity is that, is a primitive annotator > >> >> >> implented > >> >> >> >>>> >>>> >>> >> in > >> >> >> >>>> >>>> >>> >> c++, > >> >> >> >>>> >>>> >>> >> but is deployed with de deployAsyncService.sh de > >> >> uima-as > >> >> >> >>>> >>>> >>> >> directly. > >> >> >> >>>> >>>> >>> >> > >> >> >> >>>> >>>> >>> >> 2017-07-04 14:53 GMT-04:00, nelson rivera < > >> >> >> >>>> >>>> [email protected]>: > >> >> >> >>>> >>>> >>> >>> In my program main, i initialize a uimas-as > >> >> >> >>>> >>>> >>> >>> client > >> >> only > >> >> >> >>>> >>>> >>> >>> once > >> >> >> >>>> >>>> >>> >>> and > >> >> >> >>>> >>>> >>> >>> I > >> >> >> >>>> >>>> >>> >>> use > >> >> >> >>>> >>>> >>> >>> this client to consume all requests to a > >> >> >> >>>> >>>> >>> >>> displayed > >> >> >> service > >> >> >> >>>> >>>> uima-as. > >> >> >> >>>> >>>> >>> >>> > >> >> >> >>>> >>>> >>> >>> Before process a request i ask to uima-as client > >> >> >> >>>> >>>> >>> >>> with: > >> >> >> >>>> >>>> >>> >>> BaseUIMAAsynchronousEngine_ > impl.connectionOpen() > >> >> >> >>>> >>>> >>> >>> and if the result es "false", i stop the uima-as > >> >> client > >> >> >> >>>> >>>> >>> >>> and > >> >> >> >>>> >>>> >>> >>> initialize > >> >> >> >>>> >>>> >>> >>> it > >> >> >> >>>> >>>> >>> >>> again. > >> >> >> >>>> >>>> >>> >>> > >> >> >> >>>> >>>> >>> >>> With some steps, still undefined, but after > >> >> >> >>>> >>>> >>> >>> restart > >> >> the > >> >> >> >>>> UIMA-AS > >> >> >> >>>> >>>> >>> >>> Broker > >> >> >> >>>> >>>> >>> >>> without restart service uima-as. > >> >> >> >>>> >>>> >>> >>> The instruccion to uima-as client > >> >> >> >>>> >>>> >>> >>> BaseUIMAAsynchronousEngine_ > impl.connectionOpen() > >> >> return > >> >> >> >>>> "true", > >> >> >> >>>> >>>> but > >> >> >> >>>> >>>> >>> >>> them i get a "javax.jms.IllegalStateException: > >> >> >> >>>> >>>> >>> >>> The > >> >> >> Session > >> >> >> >>>> is > >> >> >> >>>> >>>> >>> >>> closed" > >> >> >> >>>> >>>> >>> >>> when proces the request. > >> >> >> >>>> >>>> >>> >>> > >> >> >> >>>> >>>> >>> >>> Wich is the best way to know when connections > >> >> >> >>>> >>>> >>> >>> betewen > >> >> >> >>>> >>>> >>> >>> cliente > >> >> >> >>>> >>>> >>> >>> uima-as > >> >> >> >>>> >>>> >>> >>> and ActiveMQ is broken? and i should restart > >> uima-as > >> >> >> >>>> >>>> >>> >>> client. > >> >> >> >>>> >>>> >>> >>> > >> >> >> >>>> >>>> >>> >> > >> >> >> >>>> >>>> >>> > > >> >> >> >>>> >>>> >>> > >> >> >> >>>> >>>> >> > >> >> >> >>>> >>>> >> > >> >> >> >>>> >>>> > > >> >> >> >>>> >>>> > >> >> >> >>>> >>> > >> >> >> >>>> >>> > >> >> >> >>>> >> > >> >> >> >>>> > > >> >> >> >>>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >> > >> >> >> > > >> >> >> > >> >> > > >> >> > >> > > >> > > >
