[
https://issues.apache.org/jira/browse/CASSANDRA-19831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Isaac Philip Liu updated CASSANDRA-19831:
-----------------------------------------
Test and Documentation Plan:
-No functional/behavioral changes introduced in this patch. Changes one line
(log message) in JVMStabilityInspector. JVMStabilityInspectorTest unit tests
testForceHeapSpaceOom and testForceHeapSpaceOomExclude does not assert on log
content so test behavior is also unaffected.-
Only behavioural change is changing casing for "GC Overhead limit exceeded" ->
"GC overhead limit exceeded" in the ignore set, covered by
testForceHeapSpaceOomExclude()
was:No functional/behavioral changes introduced in this patch. Changes one
line (log message) in JVMStabilityInspector. JVMStabilityInspectorTest unit
tests testForceHeapSpaceOom and testForceHeapSpaceOomExclude does not assert on
log content so test behavior is also unaffected.
> Improve JVMStabilityInspector error message to clarify off-heap vs heap OOM
> errors
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-19831
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19831
> Project: Apache Cassandra
> Issue Type: Task
> Components: Local/Other
> Reporter: Brad Schoening
> Assignee: Isaac Philip Liu
> Priority: Low
> Attachments: ci_summary.html, results_details.tar.xz
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> When we see Direct buffer memory errors, the JVMStabilityInspector appears to
> suggest it's a heap OOM to casual users. This can confuse users who think
> that increasing the heap may help solve the problem. A vendors support staff
> incorrectly suggested increasing the heap in this scenario.
> In inspectThrowable(), when t is a non-heap OOM (e.g. Direct buffer memory),
> forceHeapSpaceOomMaybe(t) throws a brand-new JVM-generated OutOfMemoryError:
> Java heap space, and that error propagates up the stack. The last, most
> prominent stack trace the user sees says Java heap space, with a stack
> pointing into forceHeapSpaceOomMaybe — not into Bits.reserveMemory. The
> synthetic OOM doesn't have a direct reference to the original error. The real
> cause is only findable by scrolling up in the log to the terse "Force heap
> space OutOfMemoryError in the presence of" line.
> A more clear error message could be something like:
> Off-heap OOM forcing heap space OutOfMemoryError after OutOfMemoryError:
> Direct buffer memory
> {quote}2024-08-01 17:00:00,16 [ERROR] [Messaging-EventLoop-3-6]
> cluster_id=001 ip_address=172.0.0.1 JVMStabilityInspector.java:151 - Force
> heap space OutOfMemoryError in the presence of
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
> at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
> at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318)
> at org.apache.cassandra.utils.MerkleTree.allocate(MerkleTree.java:742)
> at
> org.apache.cassandra.utils.MerkleTree.deserializeOffHeap(MerkleTree.java:780)
> {quote}
> The comment in the code is clear, but
> {quote} /**
> * Intentionally produce a heap space OOM upon seeing a non heap memory
> OOM.
> * Direct buffer OOM cannot trigger JVM OOM error related options,
> * e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
> * See CASSANDRA-15214 and CASSANDRA-17128 for more details
> */
> {quote}
> but the log message users see begins with 'Force heap space
> OutOfMemoryError', which suggests a heap memory issue, not off-heap memory.
> Very confusing for general users and junior support staff.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
