We have an Artemis broker setup running as stateful set with 2 replicas in K8s.
Each replica runs with 6000 MB of heap and 9000 MB of container memory limit.
We have some 300K pending messages on a given address and total persistent size
of it is 1.5 GB on ss-0. Due to some unforeseen issue, we need to purge these
messages. Whenever we try to purge the messages using broker GUI via operations
with a flush limit of 10 or 100 or 1000, ss-0 container gets restarted. While
browsing and purging works fine but it has limited utility as we can't delete
300k messages that way. But using operations to purge is making ss-0 restart.
We have captured following stat from ss-0 pod before starting purge via
operations. Any suggestion to fix this issue? Is it a known issue? If we need
to review or tweak any memory setting?
Memory utilization from broker GUI JMX --> java.lang --> Memory
Heap memory usage { "init": 6291456000, "committed": 6291456000, "max":
6291456000, "used": 4304010768 }
Non heap memory usage { "init": 7667712, "committed": 134938624,
"max": 1224736768, "used": 130441792 }
Object name java.lang:type=Memory
Direct memory utilization from broker GUI JMX --> java.nio --> BufferPool
Count Memory used Name Total capacity Object name
0 0 mapped - 'non-volatile memory' 0
java.nio:name=mapped - 'non-volatile memory',type=BufferPool
163 46903209 direct 46903208
java.nio:name=direct,type=BufferPool
26 274632 mapped 274632
java.nio:name=mapped,type=BufferPool
Container memory capture via k8s top pod command
> kubectl -n xxx top pod xxxx-ss-0 --containers
POD NAME CPU(cores) MEMORY(bytes)
xxxx-ss-0 abcd 2998m 8322Mi
Native memory summary of broker pod
> kubectl -n xxx exec xxxx-ss-0 -- jcmd 1 VM.native_memory summary
Native Memory Tracking:
(Omitting categories weighting less than 1KB)
Total: reserved=7354797KB, committed=6646793KB
malloc: 99469KB #456441
mmap: reserved=7255328KB, committed=6547324KB
- Java Heap (reserved=6144000KB, committed=6144000KB)
(mmap: reserved=6144000KB, committed=6144000KB)
- Class (reserved=427709KB, committed=9213KB)
(classes #12081)
( instance classes #11254, array classes #827)
(malloc=1725KB #39470) (at peak)
(mmap: reserved=425984KB, committed=7488KB)
( Metadata: )
( reserved=65536KB, committed=59648KB)
( used=59166KB)
( waste=482KB =0.81%)
( Class space:)
( reserved=425984KB, committed=7488KB)
( used=7092KB)
( waste=396KB =5.29%)
- Thread (reserved=95220KB, committed=18452KB)
(thread #172)
(stack: reserved=94724KB, committed=17956KB)
(malloc=298KB #1032) (peak=319KB #1075)
(arena=198KB #340) (peak=757KB #256)
- Code (reserved=251264KB, committed=48728KB)
(malloc=3576KB #16397) (peak=3576KB #16398)
(mmap: reserved=247688KB, committed=45152KB)
- GC (reserved=282161KB, committed=282161KB)
(malloc=21193KB #31118) (peak=22768KB #31692)
(mmap: reserved=260968KB, committed=260968KB)
- Compiler (reserved=1470KB, committed=1470KB)
(malloc=1306KB #1820) (peak=1321KB #1814)
(arena=165KB #5) (peak=45072KB #14)
- Internal (reserved=1059KB, committed=1059KB)
(malloc=1023KB #23952) (at peak)
(mmap: reserved=36KB, committed=36KB)
- Other (reserved=44711KB, committed=44711KB)
(malloc=44711KB #204) (peak=45370KB #195)
- Symbol (reserved=12165KB, committed=12165KB)
(malloc=10175KB #299383) (peak=10199KB #299791)
(arena=1990KB #1) (at peak)
- Native Memory Tracking (reserved=7234KB, committed=7234KB)
(malloc=102KB #1764) (peak=103KB #1769)
(tracking overhead=7132KB)
- Shared class space (reserved=16384KB, committed=12068KB)
(mmap: reserved=16384KB, committed=12068KB)
- Arena Chunk (reserved=179KB, committed=179KB)
(malloc=179KB #411) (peak=47415KB #1421)
- Module (reserved=275KB, committed=275KB)
(malloc=275KB #1894) (at peak)
- Safepoint (reserved=8KB, committed=8KB)
(mmap: reserved=8KB, committed=8KB)
- Synchronization (reserved=167KB, committed=167KB)
(malloc=167KB #1578) (peak=171KB #1588)
- Serviceability (reserved=1KB, committed=1KB)
(malloc=1KB #6) (at peak)
- Metaspace (reserved=65911KB, committed=60023KB)
(malloc=375KB #355) (at peak)
(mmap: reserved=65536KB, committed=59648KB)
- String Deduplication (reserved=4381KB, committed=4381KB)
(malloc=4381KB #34592) (peak=6551KB #38671)
- Object Monitors (reserved=497KB, committed=497KB)
(malloc=497KB #2448) (peak=572KB #2814)
Thanks
Shiv
