Hi,
We use derby as one option for database support for an enterprise
server product. We’re using derby 10.2.1.6 with the 1.5.0_11 jre,
hotspot server version. Occasionally when our application is using
derby (it also supports oracle, sql server, and never seems to crash
when using those) we get crashes. The jvm will dump hs_err file in the
working directory. I’ve got a bunch of these sample files output below
(just the beginning of the output).
We cannot trace down a particular action within our application that
causes these, but low memory, VM environments seem to trigger it.
Sometimes right at startup, sometimes after running for a while. Our
application has a fixed upper bounded jvm memory usage of 1 GB
(-Xmx1024m), but the VM’s never have that much dedicated ram. However,
we don’t get OutOfMemory exceptions or anything anomalous leading up
to these crashes, they just happen.
The crashes are not always in a “CompilerThread” (crashes 1-3 below),
sometimes they are in our code too (see crash 4 below). And then
sometimes they’re not even in derby code (crash 5 and 6 below).
Below are a few sample crashes. Does anyone have **any** suggests on
what I can try to do here? I don’t know where to begin!!!
--- CRASH1 ---
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_INT_DIVIDE_BY_ZERO (0xc0000094) at pc=0x6dac8e7e,
pid=1600, tid=1880
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_11-b03 mixed mode)
# Problematic frame:
# V [jvm.dll+0x1e8e7e]
#
--------------- T H R E A D ---------------
Current thread (0x47f58e90): JavaThread "CompilerThread0" daemon
[_thread_in_native, id=1880]
siginfo: ExceptionCode=0xc0000094
Registers:
EAX=0x00000000, EBX=0x48766a38, ECX=0x00000000, EDX=0x00000000
ESP=0x481ef0b8, EBP=0x488adfa4, ESI=0x00000000, EDI=0x00000015
EIP=0x6dac8e7e, EFLAGS=0x00010246
Top of Stack: (sp=0x481ef0b8)
0x481ef0b8: 481ef360 49edea80 481ef0f0 48766a78
0x481ef0c8: 49edea80 6dacc260 481ef360 49ee02c0
0x481ef0d8: 48763b14 48763b14 481ef360 48766a78
0x481ef0e8: 48766a38 48766a78 49edea80 6dacc48a
0x481ef0f8: 00000001 481ef134 49ee02c0 481ef360
0x481ef108: 4aafd408 00000000 6dacfa74 481ef360
0x481ef118: 481ef134 0000002a 481ef7d0 00000003
0x481ef128: 00000000 00000003 47f59020 47f59020
Instructions: (pc=0x6dac8e7e)
0x6dac8e6e: f8 77 37 8b f7 0f af f1 3b f0 77 2e 8b c6 33 d2
0x6dac8e7e: f7 f1 3b f8 75 24 8b 44 24 18 8b 15 3c 6e bf 6d
Stack: [0x481a0000,0x481f0000), sp=0x481ef0b8, free space=316k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
V [jvm.dll+0x1e8e7e]
Current CompileTask:
opto:1825
org.apache.derby.impl.services.locks.LockControl.getLock(Ljava/lang/Object;Ljava/lang/Object;)Lorg/apache/derby/impl/services/locks/Lock;
(102 bytes)
--- CRASH2 ---
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_INT_DIVIDE_BY_ZERO (0xc0000094) at pc=0x6dac8e7e,
pid=1132, tid=1572
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_11-b03 mixed mode)
# Problematic frame:
# V [jvm.dll+0x1e8e7e]
#
--------------- T H R E A D ---------------
Current thread (0x47f099b8): JavaThread "CompilerThread0" daemon
[_thread_in_native, id=1572]
siginfo: ExceptionCode=0xc0000094
Registers:
EAX=0x00000000, EBX=0x49c18720, ECX=0x00000000, EDX=0x00000000
ESP=0x481ef2b0, EBP=0x4920afa0, ESI=0x00000000, EDI=0x00000075
EIP=0x6dac8e7e, EFLAGS=0x00010246
Top of Stack: (sp=0x481ef2b0)
0x481ef2b0: 481ef558 48fda4e8 481ef2e8 49c18760
0x481ef2c0: 48fda4e8 6dacc260 481ef558 48fdd428
0x481ef2d0: 490308e4 490308e4 481ef558 49c18760
0x481ef2e0: 49c18720 49c18760 48fda4e8 6dacc48a
0x481ef2f0: 00000001 481ef32c 48fdd428 481ef558
0x481ef300: 4a0afbb8 00000000 6dacfa74 481ef558
0x481ef310: 481ef32c 0000002a 481ef9c8 00000003
0x481ef320: 00000000 00000004 47f09b50 47f09b50
Instructions: (pc=0x6dac8e7e)
0x6dac8e6e: f8 77 37 8b f7 0f af f1 3b f0 77 2e 8b c6 33 d2
0x6dac8e7e: f7 f1 3b f8 75 24 8b 44 24 18 8b 15 3c 6e bf 6d
Stack: [0x481a0000,0x481f0000), sp=0x481ef2b0, free space=316k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
V [jvm.dll+0x1e8e7e]
Current CompileTask:
opto:875
org.apache.derby.impl.services.locks.LockControl.isGrantable(ZLjava/lang/Object;Ljava/lang/Object;)Z
(158 bytes)
--- CRASH3 ---
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d9796d8, pid=1124,
tid=1192
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_11-b03 mixed mode)
# Problematic frame:
# V [jvm.dll+0x996d8]
#
--------------- T H R E A D ---------------
Current thread (0x47f0a468): JavaThread "CompilerThread1" daemon
[_thread_in_native, id=1192]
siginfo: ExceptionCode=0xc0000005, reading address 0x10200694
Registers:
EAX=0x102000c0, EBX=0x00000100, ECX=0x4899fe60, EDX=0x00020000
ESP=0x4823ed00, EBP=0x00004000, ESI=0x4899f878, EDI=0x00200000
EIP=0x6d9796d8, EFLAGS=0x00010206
Top of Stack: (sp=0x4823ed00)
0x4823ed00: 00000004 4a2376c4 00000000 4823f49c
0x4823ed10: 6dadf3c5 4a2376e8 6dadf415 00000010
0x4823ed20: 4a2376c4 4899f878 4a2376c4 00000000
0x4823ed30: 4823f49c 00000000 00000004 4899f888
0x4823ed40: 00000000 00000004 6dadf3c5 4a2376c4
0x4823ed50: 4899f878 00000000 00000001 4823f49c
0x4823ed60: 4a237750 4823f3a0 00000000 00000000
0x4823ed70: 00000002 4823edac 00000000 00000003
Instructions: (pc=0x6d9796d8)
0x6d9796c8: 3f 85 b9 d4 05 00 00 74 37 8b 46 8c 85 c0 74 30
0x6d9796d8: 85 98 d4 05 00 00 74 28 8b 90 b0 00 00 00 8b 81
Stack: [0x481f0000,0x48240000), sp=0x4823ed00, free space=315k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
V [jvm.dll+0x996d8]
[error occurred during error reporting, step 120, id 0xc0000005]
Current CompileTask:
opto:2864
org.apache.derby.impl.store.raw.data.FileContainer.trackUnfilledPage(JZ)V
(20 bytes)
--- CRASH4 ---
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00bb906a, pid=1316,
tid=480
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_11-b03 mixed mode)
# Problematic frame:
# J org.apache.derby.impl.sql.compile.SQLParser.jj_3R_264()Z
#
--------------- T H R E A D ---------------
Current thread (0x48c75a08): JavaThread "JobCreationThread" daemon
[_thread_in_Java, id=480]
siginfo: ExceptionCode=0xc0000005, reading address 0xffffffff
Registers:
EAX=0x4468fdb8, EBX=0x4c90ef40, ECX=0x0af77408, EDX=0x0af77408
ESP=0x4c90ef0d, EBP=0x4c90ef64, ESI=0x5a8b7c49, EDI=0x4c90ef70
EIP=0x00bb906a, EFLAGS=0x00010202
Top of Stack: (sp=0x4c90ef0d)
0x4c90ef0d: 1c0af774 3c4c90ef 6b4c90ef 0800972b
0x4c90ef1d: be0af774 b1446904 444468ff 004c90ef
0x4c90ef2d: 64009a60 704c90ef 444c90ef 404c90ef
0x4c90ef3d: 6b4c90ef 0800972b 480af774 be4c90ef
0x4c90ef4d: 70446904 004c90ef 00446db8 c8000000
0x4c90ef5d: 70446904 904c90ef 6b4c90ef 1800972b
0x4c90ef6d: 0803f466 740af774 c94c90ef 9844690e
0x4c90ef7d: 004c90ef 00446db8 d8000000 9844690e
Instructions: (pc=0x00bb906a)
0x00bb905a: 74 24 18 89 4c 24 04 8b d1 1f be 49 7c 8b 5a 44
0x00bb906a: 8b 72 40 3b f3 74 1e 8b 5e 0c 89 6c 24 10 89 7c
Stack: [0x4c8c0000,0x4c910000), sp=0x4c90ef0d, free space=315k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
J org.apache.derby.impl.sql.compile.SQLParser.jj_3R_264()Z
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_227()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_191()Z+6
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_121()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_70()Z+161
j org.apache.derby.impl.sql.compile.SQLParser.jj_3_23()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_100()Z+43
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_56()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_3_9()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_226()Z+72
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_188()Z+18
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_116()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_68()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_67()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_3_18()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_115()Z+100
j org.apache.derby.impl.sql.compile.SQLParser.jj_3R_66()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_3_17()Z+1
j org.apache.derby.impl.sql.compile.SQLParser.jj_2_17(I)Z+19
j
org.apache.derby.impl.sql.compile.SQLParser.selectList()Lorg/apache/derby/impl/sql/compile/ResultColumnList;+90
j
org.apache.derby.impl.sql.compile.SQLParser.querySpecification()Lorg/apache/derby/impl/sql/compile/ResultSetNode;+24
j
org.apache.derby.impl.sql.compile.SQLParser.simpleTable()Lorg/apache/derby/impl/sql/compile/ResultSetNode;+33
j
org.apache.derby.impl.sql.compile.SQLParser.nonJoinQueryPrimary()Lorg/apache/derby/impl/sql/compile/ResultSetNode;+41
j
org.apache.derby.impl.sql.compile.SQLParser.nonJoinQueryTerm(Lorg/apache/derby/impl/sql/compile/ResultSetNode;I)Lorg/apache/derby/impl/sql/compile/ResultSetNode;+1
j
org.apache.derby.impl.sql.compile.SQLParser.queryExpression(Lorg/apache/derby/impl/sql/compile/ResultSetNode;I)Lorg/apache/derby/impl/sql/compile/ResultSetNode;+3
j
org.apache.derby.impl.sql.compile.SQLParser.preparableSelectStatement(Z)Lorg/apache/derby/impl/sql/compile/CursorNode;+20
j
org.apache.derby.impl.sql.compile.SQLParser.preparableSQLDataStatement()Lorg/apache/derby/impl/sql/compile/QueryTreeNode;+121
j
org.apache.derby.impl.sql.compile.SQLParser.StatementPart([Lorg/apache/derby/impl/sql/compile/Token;)Lorg/apache/derby/impl/sql/compile/QueryTreeNode;+337
j
org.apache.derby.impl.sql.compile.SQLParser.Statement(Ljava/lang/String;[Ljava/lang/Object;)Lorg/apache/derby/impl/sql/compile/QueryTreeNode;+8
j
org.apache.derby.impl.sql.compile.ParserImpl.parseGoalProduction(Ljava/lang/String;[Ljava/lang/Object;)Lorg/apache/derby/impl/sql/compile/QueryTreeNode;+11
j
org.apache.derby.impl.sql.compile.ParserImpl.parseStatement(Ljava/lang/String;[Ljava/lang/Object;)Lorg/apache/derby/impl/sql/compile/QueryTreeNode;+59
j
org.apache.derby.impl.sql.GenericStatement.prepMinion(Lorg/apache/derby/iapi/sql/conn/LanguageConnectionContext;Z[Ljava/lang/Object;Lorg/apache/derby/iapi/sql/dictionary/SchemaDescriptor;Z)Lorg/apache/derby/iapi/sql/PreparedStatement;+562
j
org.apache.derby.impl.sql.GenericStatement.prepare(Lorg/apache/derby/iapi/sql/conn/LanguageConnectionContext;Z)Lorg/apache/derby/iapi/sql/PreparedStatement;+12
j
org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Lorg/apache/derby/iapi/sql/dictionary/SchemaDescriptor;Ljava/lang/String;ZZ)Lorg/apache/derby/iapi/sql/PreparedStatement;+15
j
org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Lorg/apache/derby/impl/jdbc/EmbedConnection;Ljava/lang/String;ZIIII[I[Ljava/lang/String;)V+67
j
org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Lorg/apache/derby/impl/jdbc/EmbedConnection;Ljava/lang/String;ZIIII[I[Ljava/lang/String;)V+16
j
org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Lorg/apache/derby/impl/jdbc/EmbedConnection;Ljava/lang/String;ZIIII[I[Ljava/lang/String;)V+16
j
org.apache.derby.jdbc.Driver30.newEmbedPreparedStatement(Lorg/apache/derby/impl/jdbc/EmbedConnection;Ljava/lang/String;ZIIII[I[Ljava/lang/String;)Ljava/sql/PreparedStatement;+19
j
org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Ljava/lang/String;IIII[I[Ljava/lang/String;)Ljava/sql/PreparedStatement;+33
j
org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Ljava/lang/String;II)Ljava/sql/PreparedStatement;+11
j
org.apache.commons.dbcp.DelegatingConnection.prepareStatement(Ljava/lang/String;II)Ljava/sql/PreparedStatement;+16
j
org.apache.commons.dbcp.PoolingDriver$PoolGuardConnectionWrapper.prepareStatement(Ljava/lang/String;II)Ljava/sql/PreparedStatement;+11
j <clipclip – to avoid info leakage>
v ~StubRoutines::call_stub
V [jvm.dll+0xf98c0]
--- CRASH5 ---
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d9796d8, pid=1340,
tid=1428
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_11-b03 mixed mode)
# Problematic frame:
# V [jvm.dll+0x996d8]
#
--------------- T H R E A D ---------------
Current thread (0x47f09618): JavaThread "CompilerThread0" daemon
[_thread_in_native, id=1428]
siginfo: ExceptionCode=0xc0000005, reading address 0x028005d4
Registers:
EAX=0x02800000, EBX=0x00000100, ECX=0x008b7798, EDX=0x00020000
ESP=0x481eed80, EBP=0x00004000, ESI=0x008b71b0, EDI=0x00200000
EIP=0x6d9796d8, EFLAGS=0x00010206
Top of Stack: (sp=0x481eed80)
0x481eed80: 00000004 47fc1804 47fc11e0 481ef51c
0x481eed90: 6dadf3c5 47fc1828 6dadf415 00000010
0x481eeda0: 47fc1804 008b71b0 47fc1804 47fc11e0
0x481eedb0: 481ef51c 00000000 00000004 008b71c0
0x481eedc0: 00000000 00000004 6dadf3c5 47fc1804
0x481eedd0: 008b71b0 47fc11e0 00000001 481ef51c
0x481eede0: 47fc1a5c 481ef420 00000000 00000000
0x481eedf0: 00000002 481eee2c 00000000 00000003
Instructions: (pc=0x6d9796d8)
0x6d9796c8: 3f 85 b9 d4 05 00 00 74 37 8b 46 8c 85 c0 74 30
0x6d9796d8: 85 98 d4 05 00 00 74 28 8b 90 b0 00 00 00 8b 81
Stack: [0x481a0000,0x481f0000), sp=0x481eed80, free space=315k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
V [jvm.dll+0x996d8]
[error occurred during error reporting, step 120, id 0xc0000005]
Current CompileTask:
opto: 3 !
sun.net.www.ParseUtil.decode(Ljava/lang/String;)Ljava/lang/String;
(194 bytes)
--- CRASH6 ---
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d96e463, pid=708,
tid=1048
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_11-b03 mixed mode)
# Problematic frame:
# V [jvm.dll+0x8e463]
#
--------------- T H R E A D ---------------
Current thread (0x008cc960): VMThread [id=1048]
siginfo: ExceptionCode=0xc0000005, reading address 0x54c8c758
Registers:
EAX=0x54c8c758, EBX=0x44c8bf40, ECX=0x00000003, EDX=0x6dbd0154
ESP=0x47eaf9d4, EBP=0x44c8be90, ESI=0x44c8c368, EDI=0x00000025
EIP=0x6d96e463, EFLAGS=0x00010202
Top of Stack: (sp=0x47eaf9d4)
0x47eaf9d4: 43a302f0 6dc32ab8 00000000 0076fda0
0x47eaf9e4: 6dadc831 44c8be90 6dadc7c5 6dc32ab8
0x47eaf9f4: 6dadcb1a 6dc37a98 6db72642 6dc37a98
0x47eafa04: 6dc32ab8 0076fda0 00000000 6d9a570c
0x47eafa14: 6dc32ab8 00000000 00000001 0076fda0
0x47eafa24: 00000000 00000000 6d9a6e65 00000001
0x47eafa34: 00000000 00000001 00000002 6dc32ab8
0x47eafa44: 6dc32ab8 00000001 0076fda0 0076fda0
Instructions: (pc=0x6d96e463)
0x6d96e453: 74 08 3c 66 74 04 3c 08 75 19 8b 03 85 c0 74 13
0x6d96e463: 8b 08 83 e1 03 80 f9 03 74 09 53 e8 6d e2 16 00
Stack: [0x47e60000,0x47eb0000), sp=0x47eaf9d4, free space=318k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
V [jvm.dll+0x8e463]
VM_Operation (0x4aa5e99c): full generation collection, mode:
safepoint, requested by thread 0x48f01618
