This has happened twice now in about two months apart. See my initial report of a problem here:
http://markmail.org/search/?q=Had+a+problem+with+SYSCS_FREEZE_DATABASE+and+am+wondering+is+something+can+be+done+make+this+more+robust#query:Had%20a%20problem%20with%20SYSCS_FREEZE_DATABASE%20and%20am%20wondering%20is%20something%20can%20be%20done%20make%20this%20more%20robust+page:1+mid:rvpzp7q4aanslswf+state:results An earlier problem that I had here: http://markmail.org/search/?q=Had+Derby+10.8.2.2+lockup+%28was+Another+backup+question%29#query:Had%20Derby%2010.8.2.2%20lockup%20(was%20Another%20backup%20question)+page:1+mid:e54e5pc43qoymdja+state:results My utility code is in the first link and when run it output this: Failed to unfreeze the database: Insufficient data while reading from the network - expected a minimum of 6 bytes and received only 0 bytes. The connection has been terminated. >From looking at the time that the backup tar file was created (the utility >does freeze/zfs snapshot/unfreeze/create tar file) I can see that the tar >file did not get created until this morning when the Derby network server was >terminated, indicating that the utility was hung in the unfreeze and the above >message is because of the Network server being terminated. At this point the database was frozen and all connections to the database were blocked and the Derby network server had to be forcefully killed with a "kill -9" as a connection could not be acquired to issue the shutdown command. The derby.log showed this: The XA transaction timed out and is going to be rolled back. The transaction Xid is (4871251,1a193600a379ea74636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00). ---------------------------------------------------------------- Fri Oct 19 07:37:22 EDT 2012: Shutting down Derby engine ---------------------------------------------------------------- Fri Oct 19 07:38:20 EDT 2012 : Security manager installed using the Basic server security policy. Fri Oct 19 07:38:21 EDT 2012 : Apache Derby Network Server - 10.8.2.2 - (cp1) started and ready to accept connections on port 1527 Fri Oct 19 07:38:21 EDT 2012 : Apache Derby Network Server - 10.8.2.2 - (cp1) started and ready to accept connections on port 1527 ---------------------------------------------------------------- Fri Oct 19 07:38:49 EDT 2012: Booting Derby version The Apache Software Foundation - Apache Derby - 10.8.2.2 - (cp1): instance a816c00e-013a-78d1-1c49-000065089f97 on database directory /opt/canoga/canogaview/glassfish/databases/csemdb with class loader sun.misc.Launcher$AppClassLoader@61e63e3d Loaded from file:/opt/canoga/canogaview/glassfish/javadb/lib/derby.jar java.vendor=Sun Microsystems Inc. java.runtime.version=1.6.0_22-b04 user.dir=/ derby.system.home=/opt/canoga/canogaview/glassfish/databases Database Class Loader started - derby.database.classpath='CSEM.csemderby:PCS_V1.pcsderby' ---------------------------------------------------------------- Fri Oct 19 07:39:34 EDT 2012: Booting Derby version The Apache Software Foundation - Apache Derby - 10.8.2.2 - (cp1): instance 601a400f-013a-78d1-1c49-000065089f97 on database directory /opt/canoga/canogaview/glassfish/databases/csemrptsvrdb with class loader sun.misc.Launcher$AppClassLoader@61e63e3d Loaded from file:/opt/canoga/canogaview/glassfish/javadb/lib/derby.jar java.vendor=Sun Microsystems Inc. java.runtime.version=1.6.0_22-b04 user.dir=/ derby.system.home=/opt/canoga/canogaview/glassfish/databases Database Class Loader started - derby.database.classpath='' Loaded com.canoga.derby.fcn.NpaResultsQuery from database jar "PCS_V1"."PCSDERBY" Loaded com.canoga.derby.fcn.NpaResultsTableFunction from database jar "PCS_V1"."PCSDERBY" Loaded com.canoga.derby.fcn.NpaSamResultsQuery from database jar "PCS_V1"."PCSDERBY" Loaded com.canoga.derby.fcn.NpaSamResultsTableFunction from database jar "PCS_V1"."PCSDERBY" There is a background job that we have running that performs an UPDATE_STATISTICS that just so happens to be scheduled to be run at the same instance as the backup utility. In the second email thread above, I saw a lockup related to this when trying to perform a backup. I think this happened again however I could not get a stack trace this time as the customer killed the Derby network server process before this could be done. My analysis in that second email thread points out the pattern of the lockup. I am going to try to fix this but if anyone could look at the stack trace of the email thread: http://markmail.org/search/?q=Had+Derby+10.8.2.2+lockup+%28was+Another+backup+question%29#query:Had%20Derby%2010.8.2.2%20lockup%20(was%20Another%20backup%20question)+page:1+mid:3dy474zwlvi637bn+state:results and see if they see anything obvious, it would be greatly appreciated.
