Re: [xmlblaster] Client connection not being restored properly?
Hi David, I think your solution is to keep the server session alive by setting the session to unlimited. Further Notes: You could call xmlBlasterAccess.refreshSession() to refresh the server side session (but this can fail if the network is broken and the refresh is not reaching the server in time). If you don't want to use an unlimited server session you need to destroy the XmlBlasterAccess instance (without deleting the client queue entries) when going to DEAD (xmlBlasterAccess.disconnect(disConnectQos)). Use a disConnectQos.clearClientQueue(false) to keep the entries. Re-establish the connection with a new XmlBlasterAccess instance which then sends the existing queue entries. However - this will, i think, not work if you use a RAM,1.0 queue (no DB). As said in the last mail, XmlBlasterAccess behavior is fishy after going to DEAD, which needs to be fixed by us. We more clearly need to invalidate the XmlBlasterAccess instance in this case. And finally: If your scenario is a use case which we should support we can think of changing the client library behavior so that a server session timeout triggers a POLLING (and not a DEAD) which would solve your case. Is there any logical reason to keep the to-DEAD pattern (Michele?) regards Marcel David Kerry wrote: On Wed, Apr 02, 2008 at 07:42:54PM +0200, Marcel Ruff wrote: Hi David, if the server side session times out the expected behavior is that the client goes to dead (what you client seem to do). That your client recovers from DEAD is not intended and is considered a bug, see http://www.xmlblaster.org/xmlBlaster/doc/requirements/client.failsafe.html#failsave The transition (9) and (8) should not happen. Thanks for reporting best regards Marcel Hi Marcel, Whoops! Then I guess the original issue is still outstanding then. If my client goes from ALIVE-DEAD, simply because of a session timeout on the server side, what are my options on the client side to reconnect gracefully and to prevent the loss of the messages still queued on the client side? The client still happily accepts messages for queueing even though the connection state is DEAD and returns no errors. -- Marcel Ruff http://www.xmlBlaster.org http://watchee.net Phone: +49 7551 309371
Re: [xmlblaster] Client connection not being restored properly?
Hi Marcel, In fact, that is what I have done for now (set session to unlimited). The only reason it has been an issue is because the default client session timeout is 24 hours and in some cases, I forgot to override the default in some of our low-traffic situations. Oops. If I were to file some bugs against this it would be: 1) What is the purpose of the session timing out on the server side when the client is still connected/alive (as far as a network connection is concerned, eg - ping requests are still being answered by the client)? If the client's alive, keep the session alive and just eliminate the server-side session timeout perhaps? 2) If the client reaches the 'DEAD' state, it shouldn't accept and queue more messages for publishing that it will never/can never publish. -- David Kerry On Thu, Apr 03, 2008 at 09:32:41AM +0200, Marcel Ruff wrote: Hi David, I think your solution is to keep the server session alive by setting the session to unlimited. Further Notes: You could call xmlBlasterAccess.refreshSession() to refresh the server side session (but this can fail if the network is broken and the refresh is not reaching the server in time). If you don't want to use an unlimited server session you need to destroy the XmlBlasterAccess instance (without deleting the client queue entries) when going to DEAD (xmlBlasterAccess.disconnect(disConnectQos)). Use a disConnectQos.clearClientQueue(false) to keep the entries. Re-establish the connection with a new XmlBlasterAccess instance which then sends the existing queue entries. However - this will, i think, not work if you use a RAM,1.0 queue (no DB). As said in the last mail, XmlBlasterAccess behavior is fishy after going to DEAD, which needs to be fixed by us. We more clearly need to invalidate the XmlBlasterAccess instance in this case. And finally: If your scenario is a use case which we should support we can think of changing the client library behavior so that a server session timeout triggers a POLLING (and not a DEAD) which would solve your case. Is there any logical reason to keep the to-DEAD pattern (Michele?) regards Marcel David Kerry wrote: On Wed, Apr 02, 2008 at 07:42:54PM +0200, Marcel Ruff wrote: Hi David, if the server side session times out the expected behavior is that the client goes to dead (what you client seem to do). That your client recovers from DEAD is not intended and is considered a bug, see http://www.xmlblaster.org/xmlBlaster/doc/requirements/client.failsafe.html#failsave The transition (9) and (8) should not happen. Thanks for reporting best regards Marcel Hi Marcel, Whoops! Then I guess the original issue is still outstanding then. If my client goes from ALIVE-DEAD, simply because of a session timeout on the server side, what are my options on the client side to reconnect gracefully and to prevent the loss of the messages still queued on the client side? The client still happily accepts messages for queueing even though the connection state is DEAD and returns no errors. -- Marcel Ruff http://www.xmlBlaster.org http://watchee.net Phone: +49 7551 309371
Re: [xmlblaster] Client connection not being restored properly?
Hi David, if the server side session times out the expected behavior is that the client goes to dead (what you client seem to do). That your client recovers from DEAD is not intended and is considered a bug, see http://www.xmlblaster.org/xmlBlaster/doc/requirements/client.failsafe.html#failsave The transition (9) and (8) should not happen. Thanks for reporting best regards Marcel David Kerry wrote: On Mon, Mar 10, 2008 at 11:10:54AM +0100, Marcel Ruff wrote: Hi David, fail safe mode should also set a positive session id, like this the client queue is found again on restart and you don't need another queue to store the entries. Something like java -Dcom.sun.management.jmxremote javaclients.HelloWorldSubscribe -session.name subscriber/1 -dispatch/callback/retries -1 -connect/qos/persistent true -persistentSubscribe true The persistent settings are only needed to recover from server restart and to not loose a message if a publisher reconnects and publishes earlier, before our subscriber reconnects and re-subscribes. best regards, Marcel Hi Marcel, Something's still fishy here. Although I'm not setting a persistent queue (I'm not concerned about queued incoming messages while disconnected right now), I am setting all the other parameters for a failsafe connection (callback retries -1, positive session id). I've setup my client to be notified of connection state via the I_ConnectionStateListener interface. When I attempt to publish a message after the session times out, I get a callback when the session times out (reachedDead), which is correct. However, what's not happening reliably is the reachedPolling state. Most of the time my test client goes reachedDead - reachedPolling - reachedAlive. In some cases it just goes reachedDead and no further. I did note this exception that popped out on the client side in a case where the client didn't get to a polling state. I don't know if it's relevant or not though. I didn't see any other log output or exceptions around the time this occurred. The server side simply indicates the session timed out. 2008-04-01 13:45:36,602 INFO [STDOUT] java.lang.Exception: Stack trace 2008-04-01 13:45:36,603 INFO [STDOUT] at java.lang.Thread.dumpStack(Thread.java:1158) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.client.ClientErrorHandler.handleError(ClientErrorHandler.java:79) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.dispatch.DispatchManager.givingUpDelivery(DispatchManager.java:379) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.dispatch.DispatchManager.shutdownFomAnyState(DispatchManager.java:371) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.updateState(DispatchConnectionsHandler.java:368) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.toDead(DispatchConnectionsHandler.java:336) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.dispatch.DispatchConnection.handleTransition(DispatchConnection.java:498) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.dispatch.DispatchConnection.send(DispatchConnection.java:241) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.send(DispatchConnectionsHandler.java:455) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:70) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.dispatch.DispatchManager.putPre(DispatchManager.java:668) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.dispatch.DispatchManager.putPre(DispatchManager.java:642) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.util.queue.ram.RamQueuePlugin.put(RamQueuePlugin.java:723) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.client.XmlBlasterAccess.queueMessage(XmlBlasterAccess.java:901) 2008-04-01 13:45:36,603 INFO [STDOUT] at org.xmlBlaster.client.XmlBlasterAccess.publish(XmlBlasterAccess.java:1061) 2008-04-01 13:45:36,603 INFO [STDOUT] at com.navahonetworks.app.jboss.xmlblaster.NNXmlBlasterClientImpl.publish(NNXmlBlasterClientImpl.java:165) 2008-04-01 13:45:36,603 INFO [STDOUT] at com.navahonetworks.app.jboss.xmlblaster.NNXmlBlasterClientImpl.publish(NNXmlBlasterClientImpl.java:138) 2008-04-01 13:45:36,604 INFO [STDOUT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2008-04-01 13:45:36,604 INFO [STDOUT] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 2008-04-01 13:45:36,604 INFO [STDOUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 2008-04-01 13:45:36,604 INFO [STDOUT] at java.lang.reflect.Method.invoke(Method.java:585) 2008-04-01 13:45:36,604 INFO [STDOUT] at com.navahonetworks.ejb.session.XmlblasterLogicBean.publish(XmlblasterLogicBean.java:287) 2008-04-01 13:45:36,604 INFO
Re: [xmlblaster] Client connection not being restored properly?
On Wed, Apr 02, 2008 at 07:42:54PM +0200, Marcel Ruff wrote: Hi David, if the server side session times out the expected behavior is that the client goes to dead (what you client seem to do). That your client recovers from DEAD is not intended and is considered a bug, see http://www.xmlblaster.org/xmlBlaster/doc/requirements/client.failsafe.html#failsave The transition (9) and (8) should not happen. Thanks for reporting best regards Marcel Hi Marcel, Whoops! Then I guess the original issue is still outstanding then. If my client goes from ALIVE-DEAD, simply because of a session timeout on the server side, what are my options on the client side to reconnect gracefully and to prevent the loss of the messages still queued on the client side? The client still happily accepts messages for queueing even though the connection state is DEAD and returns no errors. -- David Kerry
Re: [xmlblaster] Client connection not being restored properly?
Hi David, fail safe mode should also set a positive session id, like this the client queue is found again on restart and you don't need another queue to store the entries. Something like java -Dcom.sun.management.jmxremote javaclients.HelloWorldSubscribe -session.name subscriber/1 -dispatch/callback/retries -1 -connect/qos/persistent true -persistentSubscribe true The persistent settings are only needed to recover from server restart and to not loose a message if a publisher reconnects and publishes earlier, before our subscriber reconnects and re-subscribes. best regards, Marcel David Kerry wrote: On Wed, Feb 27, 2008 at 06:28:59PM +0100, Marcel Ruff wrote: snip My way of working around this for now is to create yet another queue, in my application that keeps a copy of the saved messages until a connection is re-established after which it puts them back in the xmlblaster queue for publishing. This works, but it's, well, ugly. Is this the intended client behaviour in this situation? Keep in mind that I'm not concerned about subscriptions and incoming messages to this client, if that makes a difference. Hi David, what about starting the client in fail safe mode? Like this the client retries to establish a connection and sends the queued messages on reconnect, regards, Marcel Hi Marcel, I think I was mistaken. I think I am connecting in fail-safe mode, if 'fail-safe' mode means setting up the connection reconnect/ping/retry values. I am not specifying an explicit session id, however (I get a negative id back upon connect). For reference, here are the connect properties I'm using: protocol=SOCKET dispatch/connection/pingInterval=3 dispatch/connection/retries=-1 dispatch/connection/delay=5000 dispatch/connection/plugin/socket/port=7607 dispatch/connection/plugin/socket/hostname= queue/connection/defaultPlugin=RAM,1.0 The rest are whatever defaults that xmlblaster client provides. From reading the reference docs, I think this actually does enable failsafe mode, correct? -- Marcel Ruff http://www.xmlBlaster.org http://watchee.net Phone: +49 7551 309371
Re: [xmlblaster] Client connection not being restored properly?
On Mon, Feb 25, 2008 at 02:48:33PM +0100, Marcel Ruff wrote: David Kerry wrote: Hello All, I'm seeing some odd behaviour on one of my clients here and I'm not sure if it's a bug, feature, or configuration problem. Client and server are running the 1.6.2 version of code. Hi David, 1. Why is secretSessionId invalid? We would need to see the server log for this, possibly the session timed out. 2. Is the client connecting in fail safe mode? o With public session ID 0: -session.name spiderman_test/1 o -dispatch/callback/retries -1 To detect going DEAD you can register xmlBlasterAccess.registerConnectionListener() and react Kerry-like (see xmlBlaster/demo/HelloWorld4.java). Still it would be good to know why the secret sessionId got invalid. best regards Marcel, Michele Hi Marcel, Well, I figured this out - it was a bit of a misunderstanding on the session timeout parameter. I didn't look close enough and thought it was in seconds instead of milliseconds. Thus, my client's sessions kept timing out on the server side much too fast! The client wasn't connecting in fail safe mode, and I was using a connection listener to monitor the connection, which helped my figure out what was going on. However, the client behaviour when the server times out the session is still a little off, I believe. What I see happening is the server kills the client session (fine), but the client doesn't notice that fact until it tries to use the connection (ie: publish), even though it has a connection-ping setup for every 10 seconds. Once the publish happens, the client goes 'oh, my connection is dead' and then starts queuing messages. At this point, however, the client never goes into a polling mode and the only way to re-establish communications is to do a disconnect(), re-instatiate a new XmlBlasterAccess and do another connect(). This whole process also has the side-effect of erasing all the queued messages. If I try to simply call connect() again, I get an exception stating connect() rejected, you are connected already, please check your code. My way of working around this for now is to create yet another queue, in my application that keeps a copy of the saved messages until a connection is re-established after which it puts them back in the xmlblaster queue for publishing. This works, but it's, well, ugly. Is this the intended client behaviour in this situation? Keep in mind that I'm not concerned about subscriptions and incoming messages to this client, if that makes a difference. -- David Kerry
Re: [xmlblaster] Client connection not being restored properly?
David Kerry wrote: On Mon, Feb 25, 2008 at 02:48:33PM +0100, Marcel Ruff wrote: David Kerry wrote: Hello All, I'm seeing some odd behaviour on one of my clients here and I'm not sure if it's a bug, feature, or configuration problem. Client and server are running the 1.6.2 version of code. Hi David, 1. Why is secretSessionId invalid? We would need to see the server log for this, possibly the session timed out. 2. Is the client connecting in fail safe mode? o With public session ID 0: -session.name spiderman_test/1 o -dispatch/callback/retries -1 To detect going DEAD you can register xmlBlasterAccess.registerConnectionListener() and react Kerry-like (see xmlBlaster/demo/HelloWorld4.java). Still it would be good to know why the secret sessionId got invalid. best regards Marcel, Michele Hi Marcel, Well, I figured this out - it was a bit of a misunderstanding on the session timeout parameter. I didn't look close enough and thought it was in seconds instead of milliseconds. Thus, my client's sessions kept timing out on the server side much too fast! The client wasn't connecting in fail safe mode, and I was using a connection listener to monitor the connection, which helped my figure out what was going on. However, the client behaviour when the server times out the session is still a little off, I believe. What I see happening is the server kills the client session (fine), but the client doesn't notice that fact until it tries to use the connection (ie: publish), even though it has a connection-ping setup for every 10 seconds. Once the publish happens, the client goes 'oh, my connection is dead' and then starts queuing messages. At this point, however, the client never goes into a polling mode and the only way to re-establish communications is to do a disconnect(), re-instatiate a new XmlBlasterAccess and do another connect(). This whole process also has the side-effect of erasing all the queued messages. If I try to simply call connect() again, I get an exception stating connect() rejected, you are connected already, please check your code. My way of working around this for now is to create yet another queue, in my application that keeps a copy of the saved messages until a connection is re-established after which it puts them back in the xmlblaster queue for publishing. This works, but it's, well, ugly. Is this the intended client behaviour in this situation? Keep in mind that I'm not concerned about subscriptions and incoming messages to this client, if that makes a difference. Hi David, what about starting the client in fail safe mode? Like this the client retries to establish a connection and sends the queued messages on reconnect, regards, Marcel -- Marcel Ruff http://www.xmlBlaster.org http://watchee.net Phone: +49 7551 309371
Re: [xmlblaster] Client connection not being restored properly?
On Wed, Feb 27, 2008 at 06:28:59PM +0100, Marcel Ruff wrote: snip My way of working around this for now is to create yet another queue, in my application that keeps a copy of the saved messages until a connection is re-established after which it puts them back in the xmlblaster queue for publishing. This works, but it's, well, ugly. Is this the intended client behaviour in this situation? Keep in mind that I'm not concerned about subscriptions and incoming messages to this client, if that makes a difference. Hi David, what about starting the client in fail safe mode? Like this the client retries to establish a connection and sends the queued messages on reconnect, regards, Marcel Hi Marcel, I think I was mistaken. I think I am connecting in fail-safe mode, if 'fail-safe' mode means setting up the connection reconnect/ping/retry values. I am not specifying an explicit session id, however (I get a negative id back upon connect). For reference, here are the connect properties I'm using: protocol=SOCKET dispatch/connection/pingInterval=3 dispatch/connection/retries=-1 dispatch/connection/delay=5000 dispatch/connection/plugin/socket/port=7607 dispatch/connection/plugin/socket/hostname= queue/connection/defaultPlugin=RAM,1.0 The rest are whatever defaults that xmlblaster client provides. From reading the reference docs, I think this actually does enable failsafe mode, correct? -- David Kerry
Re: [xmlblaster] Client connection not being restored properly?
David Kerry wrote: Hello All, I'm seeing some odd behaviour on one of my clients here and I'm not sure if it's a bug, feature, or configuration problem. Client and server are running the 1.6.2 version of code. Hi David, 1. Why is secretSessionId invalid? We would need to see the server log for this, possibly the session timed out. 2. Is the client connecting in fail safe mode? o With public session ID 0: -session.name spiderman_test/1 o -dispatch/callback/retries -1 To detect going DEAD you can register xmlBlasterAccess.registerConnectionListener() and react Kerry-like (see xmlBlaster/demo/HelloWorld4.java). Still it would be good to know why the secret sessionId got invalid. best regards Marcel, Michele Situation: o client connects to the server and after some time, something marks the connection 'idle' or dead. o client notices this and markes the connection state as 'dead'. so far so good... o further attempts by the client to publish succeed just fine, but get pushed into the save-queue to be published when the connection comes 'back' instead of actually re-initiating a connection to the server. Basically the client sort of goes zombie on me - living dead :-) Here's a relevant chunk of logging from the client side when this situation happens, just as it gets the reply from the server while attempting to publish a message: Feb 22, 2008 2:36:02 PM org.xmlBlaster.client.protocol.socket.SocketCallbackImpl run FINEST: Receiving message 1757**E**1dummyLoginName:120370896244200*publish*sessionId:172.23.254.12-null-1203703214979--1091490067-10**user.security.authentication.accessDenied*XmlBlasterException errorCode=[user.security.authentication.accessDenied] serverSideException=true location=[Authenticate] message=[#exported Your secretSessionId is invalid, no access to spiderman_test. : ] [See URL http://www.xmlblaster.org/xmlBlaster/doc/requirements/admin.errorcodes.listing.html#user.security.authentication.accessDenied]*1248*user.security.authentication.accessDenied*spiderman_test*Authenticate*en*#exported Your secretSessionId is invalid, no access to spiderman_test.*version=1.6.2,revision=exported,os.name=Linux,os.version=2.4.20,java.vm.vendor=Sun Microsystems Inc.,java.vm.version=1.5.0_06-b05,os.arch=i386,build.timestamp=12/03/2007 03:42 PM,build.java.vendor=Sun Microsystems Inc.,build.java.version=1.4.2_06*2008-02-22 14:36:02.459*errorCode=user.security.authentication.accessDenied message=#exported Your secretSessionId is invalid, no access to spiderman_test. at org.xmlBlaster.authentication.Authenticate.check(Authenticate.java:852) at org.xmlBlaster.engine.XmlBlasterImpl.publishArr(XmlBlasterImpl.java:189) at org.xmlBlaster.util.protocol.RequestReplyExecutor.receiveReply(RequestReplyExecutor.java:408) at org.xmlBlaster.protocol.socket.HandleClient.handleMessage(HandleClient.java:231) at org.xmlBlaster.protocol.socket.HandleClient$1.run(HandleClient.java:389) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690) at java.lang.Thread.run(Thread.java:595)**transaction/*true* , checksum=false, compressed=false, type=69, byte4=0, byte5=0, version=1, requestId=dummyLoginName:120370896244200, methodName=publish, sessionId=sessionId:172.23.254.12-null-1203703214979--1091490067-10 Feb 22, 2008 2:36:02 PM org.xmlBlaster.util.protocol.RequestReplyExecutor receiveReply FINE: Receiving 'EXCEPTION' message publish(dummyLoginName:120370896244200) Feb 22, 2008 2:36:02 PM org.xmlBlaster.util.protocol.RequestReplyExecutor removeResponseListener FINE: removeResponseListener(dummyLoginName:120370896244200) done Feb 22, 2008 2:36:02 PM org.xmlBlaster.util.protocol.RequestReplyExecutor$1 incomingMessage FINE: RequestId=dummyLoginName:120370896244200: return value arrived ... Feb 22, 2008 2:36:02 PM org.xmlBlaster.util.protocol.RequestReplyExecutor requestAndBlockForReply FINE: Waking up, got response for publish(requestId=dummyLoginName:120370896244200) Feb 22, 2008 2:36:02 PM org.xmlBlaster.util.protocol.RequestReplyExecutor requestAndBlockForReply FINEST: Response for publish(dummyLoginName:120370896244200) is: errorCode=user.security.authentication.accessDenied message=#exported Your secretSessionId is invalid, no access to spiderman_test. Feb 22, 2008 2:36:02 PM org.xmlBlaster.util.dispatch.DispatchConnectionsHandler removeDispatchConnection FINER: connection:client/ejb_test1203703214006: removeDispatchConnection(ClientDispatchConnection-connection:client/ejb_test1203703214006) ... Feb 22, 2008 2:36:02 PM org.xmlBlaster.util.dispatch.DispatchConnection shutdown FINER: DispatchConnection-connection:client/ejb_test1203703214006 Entering shutdown ... Feb 22, 2008 2:36:02 PM
