[ 
https://issues.apache.org/jira/browse/DERBY-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085715#comment-17085715
 ] 

Richard N. Hillegas commented on DERBY-7075:
--------------------------------------------

The attached zips contains files with the .phd extension. I believe that this 
format is an IBM dump format. I don't have a tool to read these dumps. Can you 
give us an idea of what Derby objects are consuming the memory? Thanks.

> OutOfMemory Exceptions in derby server process
> ----------------------------------------------
>
>                 Key: DERBY-7075
>                 URL: https://issues.apache.org/jira/browse/DERBY-7075
>             Project: Derby
>          Issue Type: Bug
>          Components: Network Server
>    Affects Versions: 10.8.3.0, 10.14.2.0
>            Reporter: Vamsavardhana Reddy
>            Priority: Major
>         Attachments: javaheapdump-oom-derby-101420.zip, 
> javaheapdump-oom-derby-v10833.zip
>
>
> We have two runtime environments v7.5.4 and v7.5.3 similar in terms of OS 
> memory, java version and java launch command settings for the derby and other 
> java processes required for our application in which similar 
> projects/workloads are running.  But in one of the environments we are seeing 
> out of memory (OOM) exceptions in the derby server java process.  In this 
> environment where OOM is occurring, we even refreshed Apache derby to the 
> latest version available for Java 8 and still encountered OOM exceptions.  
> Please help resolve this OOM issue. 
>  
> Exception in thread "DRDAConnThread_39" Exception in thread 
> "DRDAConnThread_41" java/lang/OutOfMemoryError: Java heap space
>  at java/nio/CharBuffer.wrap (CharBuffer.java:373)
>  at sun/nio/cs/StreamEncoder.implWrite (StreamEncoder.java:277)
>  at sun/nio/cs/StreamEncoder.write (StreamEncoder.java:137)
>  at java/io/OutputStreamWriter.write (OutputStreamWriter.java:244)
>  at java/io/BufferedWriter.flushBuffer (BufferedWriter.java:140)
>  at java/io/PrintStream.write (PrintStream.java:550)
>  at java/io/PrintStream.print (PrintStream.java:693)
>  at java/io/PrintStream.append (PrintStream.java:1089)
>  at java/io/PrintStream.append (PrintStream.java:70)
>  at com/ibm/oti/util/Util.appendTo (Util.java:225)
>  at java/lang/Throwable.printStackTrace (Throwable.java:506)
>  at java/lang/Throwable.printStackTrace (Throwable.java:302)
>  at java/lang/Throwable.printStackTrace (Throwable.java:247)
> Exception in thread "DRDAConnThread_43" 
> java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError(?)java.lang.OutOfMemoryError:
>  Java heap space
>  at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:872)
>  at 
> java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:8java/lang/OutOfMemoryError:
>  Java heap space
>  at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:872)
>  at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:866)
>  at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:866)
>  at java/lang/Thread.uncaughtException (Thread.java:1332)
> -----------------------
> :*v7.5.4*
>  OS: RHELv8 with 4GB RAM
>  Java: IBM Java SDK v8.0.6.0
>  Apache Derby v10.8.3.3, Apache Derby v10.14.2.0
> [bceglc385/Standalone ~]# /usr/java/default/jre/bin/java -version
>  java version "1.8.0_231"
>  Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6))
>  IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 
> 20191106_432135 (JIT enabled, AOT enabled)
>  OpenJ9   - f0b6be7
>  OMR      - 18d8f94
>  IBM      - 233dfb5)
>  JCL - 20191016_01 based on Oracle jdk8u231-b10
>  
> /usr/java/default/bin/java -server -Xdump:heap+system+snap:none 
> -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.ph  
> -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1
>  
> -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt
>  -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron 
> -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar:
>  -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000 
> -XX:+HeapManagementMXBeanCompatibility 
> -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger 
> -Dderby.system.home=/usr/ironhide/var/db/staging 
> -Dcom.sun.management.jmxremote 
> -Dcom.approuter.management.jmxremote.connectorserver.port=31997 
> -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote 
> -Dcom.sun.management.jmxremote.port=1997 
> -Dcom.sun.management.jmxremote.local.only=false 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dcom.sun.management.jmxremote.authenticate=true 
> -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password
>  
> -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access
>  -Dcom.approuter.logging.filename=derbyserver.log 
> -Djava.rmi.server.hostname=9.121.242.177 
> -[Djava.net|https://slack-redir.net/link?url=http%3A%2F%2FDjava.net].preferIPv4Stack=true
>  -Dcom.approuter.sysconf.appliance=true 
> -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog 
> -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true 
> -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30 
> -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true 
> -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10 
> -Dosgi.clean=true 
> -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver 
> -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true 
> -Declipse.consoleLog=true 
> -Dcom.approuter.orcmon.purger.compressionmode=sequential 
> -DIH_ROOT=/usr/ironhide 
> -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties 
> -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar
>  
> [bceglc390/Standalone ~]# cat /proc/meminfo
>  MemTotal:        4057920 kB
>  MemFree:          152896 kB
>  Buffers:          342624 kB
>  Cached:           729112 kB
>  SwapCached:         3428 kB
>  Active:          2644068 kB
>  Inactive:        1066404 kB
>  Active(anon):    1951704 kB
>  Inactive(anon):   712696 kB
>  Active(file):     692364 kB
>  Inactive(file):   353708 kB
>  Unevictable:          20 kB
>  Mlocked:              20 kB
>  SwapTotal:       3999736 kB
>  SwapFree:        3988760 kB
>  Dirty:               720 kB
>  Writeback:             0 kB
>  AnonPages:       2636560 kB
>  Mapped:            38508 kB
>  Shmem:             25648 kB
>  Slab:             129600 kB
>  SReclaimable:      48320 kB
>  SUnreclaim:        81280 kB
>  KernelStack:        6568 kB
>  PageTables:        11116 kB
>  NFS_Unstable:          0 kB
>  Bounce:                0 kB
>  WritebackTmp:          0 kB
>  CommitLimit:     6028696 kB
>  Committed_AS:    1658244 kB
>  VmallocTotal:   34359738367 kB
>  VmallocUsed:      152396 kB
>  VmallocChunk:   34359582800 kB
>  AnonHugePages:   2433024 kB
>  HugePages_Total:       0
>  HugePages_Free:        0
>  HugePages_Rsvd:        0
>  HugePages_Surp:        0
>  Hugepagesize:       2048 kB
>  DirectMap4k:        4096 kB
>  DirectMap2M:     4190208 kB
>  
> ----------
>  *v7.5.3*
>  OS: Bedrock v5 which is custom OS based on RHELv7 with 4GB RAM
>  Java: IBM Java SDK v8.0.6.0
>  Apache Derby v10.8.3.3
>  
> [bceglc390/Standalone ~]# /usr/java/default/jre/bin/java -version
>  java version "1.8.0_231"
>  Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6))
>  IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 
> 20191106_432135 (JIT enabled, AOT enabled)
>  OpenJ9   - f0b6be7
>  OMR      - 18d8f94
>  IBM      - 233dfb5)
>  JCL - 20191016_01 based on Oracle jdk8u231-b10
>  
> /usr/java/default/bin/java -server -Xdump:heap+system+snap:none 
> -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.phd 
> -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1
>  
> -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt
>  -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron 
> -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar:
>  -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000 
> -XX:+HeapManagementMXBeanCompatibility 
> -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger 
> -Dderby.system.home=/usr/ironhide/var/db/staging 
> -Dcom.sun.management.jmxremote 
> -Dcom.approuter.management.jmxremote.connectorserver.port=31997 
> -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote 
> -Dcom.sun.management.jmxremote.port=1997 
> -Dcom.sun.management.jmxremote.local.only=false 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dcom.sun.management.jmxremote.authenticate=true 
> -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password
>  
> -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access
>  -Dcom.approuter.logging.filename=derbyserver.log 
> -Djava.rmi.server.hostname=9.124.112.240 
> -[Djava.net|https://slack-redir.net/link?url=http%3A%2F%2FDjava.net].preferIPv4Stack=true
>  -Dcom.approuter.sysconf.appliance=true 
> -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog 
> -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true 
> -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30 
> -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true 
> -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10 
> -Dosgi.clean=true 
> -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver 
> -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true 
> -Declipse.consoleLog=true 
> -Dcom.approuter.orcmon.purger.compressionmode=sequential 
> -DIH_ROOT=/usr/ironhide 
> -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties 
> -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar
>  
> [cldegdh9/Standalone log]# cat /proc/meminfo
>  MemTotal:        3872868 kB
>  MemFree:          343700 kB
>  MemAvailable:     374676 kB
>  Buffers:           33100 kB
>  Cached:           170624 kB
>  SwapCached:         3848 kB
>  Active:          2540784 kB
>  Inactive:         710092 kB
>  Active(anon):    2429812 kB
>  Inactive(anon):   621324 kB
>  Active(file):     110972 kB
>  Inactive(file):    88768 kB
>  Unevictable:          20 kB
>  Mlocked:              20 kB
>  SwapTotal:       5345272 kB
>  SwapFree:        5280760 kB
>  Dirty:              1880 kB
>  Writeback:             0 kB
>  AnonPages:       3044312 kB
>  Mapped:            33676 kB
>  Shmem:              3984 kB
>  Slab:             119980 kB
>  SReclaimable:      71120 kB
>  SUnreclaim:        48860 kB
>  KernelStack:       15116 kB
>  PageTables:        13936 kB
>  NFS_Unstable:          0 kB
>  Bounce:                0 kB
>  WritebackTmp:          0 kB
>  CommitLimit:     7281704 kB
>  Committed_AS:    1755044 kB
>  VmallocTotal:   34359738367 kB
>  VmallocUsed:           0 kB
>  VmallocChunk:          0 kB
>  HardwareCorrupted:     0 kB
>  AnonHugePages:   2885632 kB
>  ShmemHugePages:        0 kB
>  ShmemPmdMapped:        0 kB
>  HugePages_Total:       0
>  HugePages_Free:        0
>  HugePages_Rsvd:        0
>  HugePages_Surp:        0
>  Hugepagesize:       2048 kB
>  Hugetlb:               0 kB
>  DirectMap4k:      126912 kB
>  DirectMap2M:     4067328 kB



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to