Re: Connection Recovery

David Blevins Mon, 23 Nov 2009 12:33:18 -0800

This is because there is no concept of blocking connections forapplications that are being redeployed. In fact our (geronimo andopenejb) redeploy is just an undeploy followed by a deploy, so there'sno special concept propagated throughout the architecture thatsomething is only temporarily gone and will be coming back.

That said, if we added all that, you'd still have to wait the exactsame amount of time for the undeploy/deploy process plus a small shaveoff of total throughput for the synchronization we'd have to add tomake the wait-for-redeploy logic work. The bigger your app gets, thelonger the time your client spends waiting, which is probably going todrive you to splinter your app into tiny fragments which onlycomplicates things more and adds even more overhead.

This is one of those "change the problem" types of situations.Instead enable multicast discovery on your server and just let itrun. When you want to "redeploy", just boot a new server with the newversion of the app. When you get that server running to your liking-- almost everyone has some sort of initialization they like to do intheir apps -- just shut down the first server and the client willimmediately roll over to the second server. No time spent waiting onredeploy and you actually get a chance to pound on your new server abit before it goes live, so no more risk that the app doesn't quitework after the redeploy.

You can do this all on the same machine with a shell script or antscript that is only slightly fancier than your standard start, deploy,run type of script.


-David

On Nov 23, 2009, at 6:20 AM, Quintin Beukes wrote:

Hey,

One of the big motivations for shifting to OpenEJB 3.1.2 from GF was
the connection recovery built into it. My inquiry was specifically
about failed remote EJB connections because a server restarts or a
network hiccup. We've always had problems with this, especially with
networks running from surface to underground.

This seems to be working. Great job!! Thanks alot. I'm aiming for
complete recovery in the following situations.
1. System exception (any failure on the part of OpenEJB, even if the
cause was my own)
2. Dropped connections due to network
3. Dropped connections from server restart
4. Dropped connections from redeploy (basically the same as (3) Iassume).
Though I have a query though. Every time I redeploy, for ex, I have to
manually go and restart all services because it takes very long for
OpenEJB client to notice something is wrong and recover.

Here is an example: If I undeploy, I get something like the following
on a client.

--------- SNIP -----------

javax.ejb.EJBException: Container has suffered a SystemException
org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:178)org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
   $Proxy2.captureLamp(Unknown Source)
net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)
java.rmi.RemoteException: The server has encountered a fatal error: No
such deployment java.rmi.RemoteException: No deployment:
VDS-lamps-ejb-3.0.jar/LampCaptureBean; nested exception is:
java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/LampCaptureBeanorg.apache.openejb.server.ejbd.EjbRequestHandler.replyWithFatalError(EjbRequestHandler.java:425)org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:81)org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149)
   org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213)org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
   org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
   org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
   org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   java.lang.Thread.run(Thread.java:619)
java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/LampCaptureBeanorg.apache.openejb.server.ejbd.EjbDaemon.getDeployment(EjbDaemon.java:191)org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:79)org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149)
   org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213)org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
   org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
   org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
   org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   java.lang.Thread.run(Thread.java:619)


23 Nov 2009 15:59:20,039 ERROR -- Exception: Service
ConfigurationService failed for instance
ConfigurationService[/10.0.0.25] (java.lang.Exception)
java.lang.Exception: ConfigurationService has an unknown failure while
servicing a request.
net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:84)net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)
javax.ejb.EJBException: Container has suffered a SystemException
org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:178)org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
   $Proxy2.captureLamp(Unknown Source)
net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)
java.rmi.RemoteException: The server has encountered a fatal error: No
such deployment java.rmi.RemoteException: No deployment:
VDS-lamps-ejb-3.0.jar/LampCaptureBean; nested exception is:
java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/LampCaptureBeanorg.apache.openejb.server.ejbd.EjbRequestHandler.replyWithFatalError(EjbRequestHandler.java:425)org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:81)org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149)
   org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213)org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
   org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
   org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
   org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   java.lang.Thread.run(Thread.java:619)
java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/LampCaptureBeanorg.apache.openejb.server.ejbd.EjbDaemon.getDeployment(EjbDaemon.java:191)org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:79)org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149)
   org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213)org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
   org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
   org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
   org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   java.lang.Thread.run(Thread.java:619)

--------- SNIP -----------

Then, for a while, even after the EJB service started up completely
and even though I'm creating new InitialContext instances I get the
following. This isn't too bad, though it does take a few minutes
(between 3 and 5) to recover.

--------- SNIP -----------
23 Nov 2009 16:00:24,802 TRACE -- Received packet[R0010088220000000000000000]
23 Nov 2009 16:00:24,804 ERROR -- Exception: Unknown failure servicing
the request (javax.ejb.EJBException)
javax.ejb.EJBException: Unknown Container Exception:
java.rmi.RemoteException: Received invalid response code from server:
-1
org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:184)org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
   $Proxy2.captureLamp(Unknown Source)
net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)
java.rmi.RemoteException: Received invalid response code fromserver: -1org.apache.openejb.client.EJBObjectHandler.businessMethod(EJBObjectHandler.java:239)org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:157)org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
   $Proxy2.captureLamp(Unknown Source)
net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)
23 Nov 2009 16:00:24,804 ERROR -- Exception: Service
ConfigurationService failed for instance
ConfigurationService[/10.0.0.25] (java.lang.Exception)
java.lang.Exception: ConfigurationService has an unknown failure while
servicing a request.
net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:84)net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)
javax.ejb.EJBException: Unknown Container Exception:
java.rmi.RemoteException: Received invalid response code from server:
-1
org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:184)org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
   $Proxy2.captureLamp(Unknown Source)
net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)
java.rmi.RemoteException: Received invalid response code fromserver: -1org.apache.openejb.client.EJBObjectHandler.businessMethod(EJBObjectHandler.java:239)org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:157)org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
   $Proxy2.captureLamp(Unknown Source)
net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)
--------- SNIP -----------

After it recovers from the above, all is back to normal.

This is quite serious. How I can get around this, even if I have to
make some code changes?

Thanks again. I'm so relieved this problem is finally over. Previously
I've had to do some serious recovery just because a client was left
running and Glassfish's client library opened so many file descriptors
for each failure that the server crashed with a No Space Left on
Device error.

Quintin Beukes

Re: Connection Recovery

Reply via email to