It happens in NIO and NIO2 using JSSE. I do not yet know, whether it
happens for other JDK vendors or versions. One VM made enough progress
to read tcnative based tests and it happens there too.
I have not seen this before, at least I do not remember.
It does happen for SLES 11, 12, 15 and RHEL 6, 7, 8 and 9. I do not
observe this on Solaris 10 or 11, but those have more system memory. The
Linux VMs typically run with 4 GB OS memory. The JVM heap seems to be
256MB.
Eclipse MAT points to the following Queue, which contains more than
90.000 nodes (shown with incoming references):
Class Name | Shallow Heap | Retained Heap
----------------------------------------------------------------------------------------------------------------------
java.util.concurrent.LinkedBlockingQueue @ 0xf1945eb0 |
48 | 204.135.592
'- messages
org.apache.tomcat.websocket.TesterMessageCountClient$BasicText @
0xf1945e88| 24 | 204.135.696
|- textMessageHandler org.apache.tomcat.websocket.WsSession @
0xf193d9a0 | 152 | 204.195.664
'- textMsgHandler org.apache.tomcat.websocket.WsFrameClient @
0xf193db00 | 120 | 41.984
----------------------------------------------------------------------------------------------------------------------
I see this stack sometimes, but it could be just a victim of the memory
shortage:
Exception in thread "WebSocketClient-SecureIO-1"
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at
com.sun.crypto.provider.GaloisCounterMode.decrypt(GaloisCounterMode.java:537)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:782)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:667)
at com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
at javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:824)
at javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
at javax.crypto.Cipher.doFinal(Cipher.java:2463)
at
sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1880)
at
sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
at
sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
at
sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109)
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:588)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:544)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:411)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:390)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:626)
at
org.apache.tomcat.websocket.AsyncChannelWrapperSecure$ReadTask.run(AsyncChannelWrapperSecure.java:269)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
JVM size info:
JVM version is 25.442-b06
using thread-local object allocation.
Parallel GC with 2 thread(s)
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 268435456 (256.0MB)
NewSize = 20971520 (20.0MB)
MaxNewSize = 89128960 (85.0MB)
OldSize = 41943040 (40.0MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 30408704 (29.0MB)
used = 30408704 (29.0MB)
free = 0 (0.0MB)
100.0% used
From Space:
capacity = 29360128 (28.0MB)
used = 0 (0.0MB)
free = 29360128 (28.0MB)
0.0% used
To Space:
capacity = 29360128 (28.0MB)
used = 0 (0.0MB)
free = 29360128 (28.0MB)
0.0% used
PS Old Generation
capacity = 179306496 (171.0MB)
used = 178862408 (170.57648468017578MB)
free = 444088 (0.42351531982421875MB)
99.75233022232501% used
10301 interned Strings occupying 923704 bytes.
Heap histogram:
num #instances #bytes class name
----------------------------------------------
1: 117372 201481240 [C
2: 117231 2813544 java.lang.String
3: 96658 2319792
java.util.concurrent.LinkedBlockingQueue$Node
4: 1023 422040 [B
5: 3366 379696 java.lang.Class
6: 5959 190688 java.util.HashMap$Node
7: 5552 177664
java.util.concurrent.ConcurrentHashMap$Node
8: 2703 157816 [Ljava.lang.Object;
9: 1312 115456 java.lang.reflect.Method
10: 1722 87376 [I
11: 530 75552 [Ljava.util.HashMap$Node;
12: 64 50920
[Ljava.util.concurrent.ConcurrentHashMap$Node;
13: 1020 48960
org.apache.tomcat.util.modeler.AttributeInfo
14: 2897 46352 java.lang.Object
15: 621 39680 [Ljava.lang.String;
16: 896 35840 java.util.LinkedHashMap$Entry
17: 678 32544 java.util.HashMap
18: 894 28608 java.util.Hashtable$Entry
19: 512 24576
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
20: 526 21040 java.lang.ref.SoftReference
21: 9 20656 [Ljava.nio.ByteBuffer;
22: 638 20416 javax.management.MBeanAttributeInfo
23: 414 19872
org.apache.tomcat.util.modeler.OperationInfo
24: 896 19616 [Ljava.lang.Class;
25: 408 16320 java.math.BigInteger
26: 273 15288 java.lang.invoke.MemberName
27: 550 13200 java.util.LinkedList$Node
28: 513 12312
java.util.concurrent.locks.ReentrantReadWriteLock
29: 166 11952 java.util.logging.Logger
30: 280 11200 javax.management.MBeanOperationInfo
31: 341 10912 sun.misc.FDBigInteger
32: 314 10048
org.apache.tomcat.util.modeler.ParameterInfo
33: 40 9584 [Ljava.util.Hashtable$Entry;
34: 59 9120 [Z
35: 223 8920 java.lang.invoke.MethodType
36: 414 8608
[Lorg.apache.tomcat.util.modeler.ParameterInfo;
37: 262 8384 java.lang.ref.WeakReference
38: 172 8256
java.util.logging.LogManager$LoggerWeakRef
39: 513 8208
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
40: 513 8208
java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
41: 513 8208
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
42: 235 7520
java.util.concurrent.locks.ReentrantLock$NonfairSync
43: 233 7456 javax.management.MBeanParameterInfo
44: 224 7168
java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry
45: 223 7136 java.lang.invoke.LambdaForm$Name
46: 110 6600 [Ljava.lang.ref.SoftReference;
47: 103 6592 java.util.concurrent.ConcurrentHashMap
48: 261 6264 java.util.ArrayList
49: 256 6144 java.lang.Long
50: 152 6080 java.util.TreeMap$Entry
51: 188 6016
com.sun.org.apache.xerces.internal.xni.QName
52: 248 5952 javax.management.ImmutableDescriptor
53: 183 5856 java.util.LinkedList
54: 145 5800 java.util.WeakHashMap$Entry
55: 89 5696 java.net.URL
56: 70 5600 java.lang.reflect.Constructor
57: 84 5560 [Ljavax.management.MBeanAttributeInfo;
58: 52 5312 [Ljava.util.WeakHashMap$Entry;
59: 216 5184
java.util.concurrent.CopyOnWriteArrayList
60: 154 4928
com.sun.jmx.mbeanserver.ConvertingMethod
61: 307 4912 java.lang.Integer
62: 13 4888 java.lang.Thread
63: 84 4704 java.lang.Package
64: 12 4608
org.apache.tomcat.util.threads.TaskThread
65: 74 4144
org.apache.tomcat.util.modeler.ManagedBean
66: 164 4120 [Ljavax.management.MBeanParameterInfo;
67: 256 4096 java.lang.Byte
68: 256 4096 java.lang.Short
69: 125 4000 java.lang.invoke.DirectMethodHandle
70: 83 3984 javax.management.MBeanInfo
71: 70 3920 java.util.LinkedHashMap
72: 244 3904 java.util.HashMap$Values
73: 160 3840
org.apache.juli.ClassLoaderLogManager$LogNode
74: 235 3760
java.util.concurrent.locks.ReentrantLock
75: 75 3600 java.lang.invoke.LambdaForm
76: 80 3520 [Ljava.lang.invoke.LambdaForm$Name;
77: 60 3360 java.lang.invoke.MethodTypeForm
78: 165 3288
[Lcom.sun.jmx.mbeanserver.MXBeanMapping;
79: 49 3136
javax.management.openmbean.OpenMBeanAttributeInfoSupport
80: 130 3120 java.time.LocalDateTime
81: 111 2976
[Ljavax.management.ObjectName$Property;
82: 29 2944 [Ljava.lang.invoke.MethodHandle;
83: 22 2880 [S
84: 10 2848
[Lcom.sun.org.apache.xerces.internal.xni.QName;
85: 17 2816 [J
86: 117 2808 javax.management.ObjectName$Property
87: 70 2800 java.io.ObjectStreamField
88: 74 2776 [Ljavax.management.MBeanOperationInfo;
89: 48 2688 java.util.ResourceBundle$CacheKey
90: 110 2640
java.lang.invoke.LambdaForm$NamedFunction
91: 64 2560 java.lang.ref.Finalizer
92: 49 2352 java.util.WeakHashMap
93: 48 2304
java.util.ResourceBundle$BundleReference
94: 48 2304 org.apache.tomcat.util.buf.ByteChunk
95: 56 2240 javax.management.ObjectName
96: 69 2208 java.lang.ref.ReferenceQueue
97: 15 2160 java.text.DecimalFormat
98: 53 2120 sun.security.util.NamedCurve
99: 128 2048 java.lang.Character
100: 13 2016 [Ljava.lang.reflect.Method;
101: 36 2016
org.apache.tomcat.util.res.StringManager$1
102: 41 1968 org.apache.tomcat.util.buf.CharChunk
103: 40 1920
org.apache.tomcat.util.buf.MessageBytes
104: 48 1920
sun.management.DiagnosticCommandArgumentInfo
105: 40 1920
sun.util.locale.LocaleObjectCache$CacheEntry
106: 75 1800 java.security.Provider$ServiceKey
107: 32 1792 java.security.Provider$Service
108: 32 1792 sun.misc.URLClassPath$JarLoader
109: 53 1696 java.security.spec.EllipticCurve
110: 68 1632 java.time.LocalDate
111: 16 1624 [[Ljava.lang.String;
112: 66 1584
sun.reflect.generics.tree.SimpleClassTypeSignature
113: 14 1568 java.util.GregorianCalendar
114: 48 1536
java.util.ResourceBundle$LoaderReference
115: 16 1536 java.util.jar.JarFile$JarFileEntry
116: 27 1512 sun.nio.cs.UTF_8$Encoder
117: 3 1488 [[B
118: 23 1472 java.util.jar.JarFile
119: 30 1440 java.time.temporal.ValueRange
120: 36 1440 java.util.PropertyResourceBundle
121: 88 1408 org.apache.juli.logging.DirectJDKLog
122: 1 1376 [Lsun.misc.FDBigInteger;
123: 28 1344 java.nio.HeapByteBuffer
124: 14 1344 sun.util.calendar.Gregorian$Date
125: 55 1320 com.sun.jmx.mbeanserver.NamedObject
126: 54 1296 java.security.spec.ECPoint
127: 32 1280 java.security.AccessControlContext
128: 66 1248
[Lsun.reflect.generics.tree.TypeArgument;
129: 26 1248 java.util.TreeMap
130: 50 1200 java.lang.Class$AnnotationData
131: 30 1200 java.time.temporal.ChronoField
132: 38 1184 [Ljava.math.BigInteger;
133: 74 1184
[Lorg.apache.tomcat.util.modeler.NotificationInfo;
134: 37 1184
java.lang.invoke.BoundMethodHandle$Species_L
135: 37 1184 java.util.regex.Pattern$Curly
136: 49 1176 org.apache.coyote.ActionCode
137: 71 1136 java.lang.ref.ReferenceQueue$Lock
138: 23 1104 sun.management.DiagnosticCommandInfo
139: 19 1064 javax.management.openmbean.ArrayType
140: 33 1056 java.io.File
141: 66 1056
sun.reflect.generics.tree.ClassTypeSignature
Regards,
Rainer