[ 
https://issues.apache.org/jira/browse/CASSANDRA-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16433463#comment-16433463
 ] 

Jürgen Albersdorfer commented on CASSANDRA-14239:
-------------------------------------------------

Had again to join a new node - giving it 72GB of Heap - caused again OOM.

I have a GC Log this time. For me, this smells strong like a Memory Leak.

Throw the attached [^gc.log.0.current.zip] against 
[http://gceasy.io|http://gceasy.io/] and you will immediatelly see what I mean.

This Node has a fast 1TB SSD, I didn't change
# memtable_flush_writers: 2
and also left
# memtable_heap_space_in_mb: 1048# memtable_offheap_space_in_mb: 1048
defaulting to 25% of Heap.

I cannot see any IO Pressure on the System during the whole bootstrap Process:
{code:java}
----system---- ---load-avg--- ---procs--- ------memory-usage----- ---paging-- 
-dsk/total- ---system-- ----total-cpu-usage---- --io/total- -net/total-
     time     | 1m   5m  15m |run blk new| used  buff  cach  free|  in   out | 
read  writ| int   csw |usr sys idl wai hiq siq| read  writ| recv  send
10-04 16:32:07| 118  128  133| 44 8.3 0.8|78.2G    0  15.9G  314M|   0     0 | 
200k 4458k|  23k   11k| 59   1  40   0   0   0|31.7  14.0 |3198k   79k
10-04 16:32:17|99.7  124  132|1.0   0 0.9|78.2G    0  15.9G  310M|   0     0 |  
 0  3123B|1509   214 |  6   0  94   0   0   0|   0  0.50 | 176k 3337B
10-04 16:32:27|84.5  120  130|1.0   0 0.8|78.2G    0  15.9G  315M|   0     0 |  
 0     0 |2312   203 |  6   0  94   0   0   0|   0     0 | 905k   10k
10-04 16:32:37|71.7  116  129|1.0   0 0.8|78.2G    0  15.9G  316M|   0     0 |  
 0   121k|1259   198 |  6   0  94   0   0   0|   0  1.20 |1737B  505B
10-04 16:32:47|60.8  112  127|1.0   0 0.8|78.2G    0  15.9G  316M|   0     0 |  
 0    37k|1240   184 |  6   0  94   0   0   0|   0  2.20 |1450B  308B
10-04 16:32:57|51.6  109  126|1.1   0 0.8|78.2G    0  15.9G  315M|   0     0 |  
 0     0 |1240   175 |  6   0  94   0   0   0|   0     0 |1541B  308B
10-04 16:33:07|43.8  105  125|1.0   0 0.8|78.2G    0  15.9G  316M|   0     0 |  
 0     0 |1218   153 |  6   0  94   0   0   0|   0     0 |1791B  593B
10-04 16:33:17|37.2  102  123|1.0   0 0.8|78.2G    0  15.9G  316M|   0     0 |  
 0    21k|1198   141 |  6   0  94   0   0   0|   0  1.40 |1496B  389B
10-04 16:33:27|31.7 98.5  122|1.0   0 0.8|78.2G    0  15.9G  316M|   0     0 |  
 0     0 |1188   122 |  6   0  94   0   0   0|   0     0 |1610B  425B
10-04 16:33:37|27.0 95.3  121|1.0   0 0.8|78.2G    0  15.9G  316M|   0     0 |  
 0     0 |1176   121 |  6   0  94   0   0   0|   0     0 |1723B  313B
10-04 16:33:47|23.0 92.2  119|1.0   0 0.9|78.2G    0  15.9G  317M|   0     0 |  
 0   307B|1165   120 |  6   0  94   0   0   0|   0  0.40 |1515B  276B
10-04 16:33:57|19.6 89.2  118|1.1   0 0.8|78.2G    0  15.9G  317M|   0     0 |  
 0     0 |1166   116 |  6   0  94   0   0   0|   0     0 |1543B  384B
10-04 16:34:07|16.7 86.3  117|1.0   0 0.8|78.2G    0  15.9G  317M|   0     0 |  
 0     0 |1169   114 |  6   0  94   0   0   0|   0     0 |1635B  582B
10-04 16:34:17|15.3 83.7  116| 12   0 1.7|78.2G    0  15.9G  312M|   0     0 |  
20k 1382B|  20k 1648 | 58   0  42   0   0   0|1.50  0.50 | 102k 7651B
10-04 16:34:27|29.9 84.5  116| 87   0 5.7|78.2G    0  15.9G  315M|   0     0 | 
248k 5055k|  40k   27k| 96   1   3   0   0   0|37.1  18.3 |4296k  424k
10-04 16:34:37|47.9 86.6  116|148 0.3 0.8|78.2G    0  15.9G  309M|   0     0 | 
232k 2647k|  35k   29k| 98   1   1   0   0   0|33.3  7.20 |2510k  207k
10-04 16:34:47|44.6 84.6  115| 24   0 1.3|78.2G    0  15.9G  310M|   0     0 | 
894k   17M|  80k   83k| 91   4   4   0   0   2| 119  59.8 |  15M 3217k
10-04 16:34:57|41.0 82.5  114| 19   0 1.0|78.2G    0  15.9G  301M|   0     0 | 
304k   19M|  35k 5311 | 95   2   2   0   0   1|40.4  56.1 |  17M  146k
10-04 16:35:07|37.9 80.5  113| 21   0 1.1|78.2G    0  15.9G  320M|   0     0 | 
342k   18M|  39k 5805 | 96   2   1   0   0   1|43.6  56.2 |  20M  179k
10-04 16:35:17|35.4 78.5  112| 20   0 0.9|78.2G    0  15.9G  315M|   0     0 | 
334k   18M|  34k 5770 | 96   2   2   0   0   0|42.5  54.2 |  17M   79k
10-04 16:35:27|33.3 76.7  111| 20   0 1.0|78.2G    0  15.9G  303M|   0     0 | 
290k   19M|  36k 6144 | 96   2   2   0   0   0|38.0  55.1 |  19M   83k
10-04 16:35:37|31.0 74.8  110| 18   0 0.8|78.2G    0  15.9G  305M|   0     0 | 
813k   23M|  42k 6870 | 94   2   3   0   0   1| 104  62.3 |  23M   90k
10-04 16:35:47|29.5 73.0  109| 21   0 0.8|78.2G    0  15.9G  323M|   0     0 | 
360k   18M|  35k 5955 | 96   2   2   0   0   0|45.8  51.4 |  18M   55k
10-04 16:35:57|28.4 71.3  108| 20 0.1 0.8|78.2G    0  15.9G  313M|   0     0 | 
325k   19M|  36k 6081 | 96   2   2   0   0   0|41.3  52.2 |  18M   54k
10-04 16:36:07|27.2 69.7  107| 21   0 0.8|78.2G    0  15.9G  304M|   0     0 | 
358k   18M|  36k 6036 | 95   2   3   0   0   0|45.5  50.7 |  18M   56k
10-04 16:36:17|26.3 68.1  106| 21   0 0.8|78.2G    0  15.9G  305M|   0     0 | 
344k   19M|  35k 6063 | 96   2   2   0   0   0|45.5  52.9 |  18M   58k
10-04 16:36:27|25.5 66.5  105| 19 0.1 0.8|78.2G    0  15.9G  301M|   0     0 | 
380k   17M|  36k 5980 | 95   2   3   0   0   0|48.7  46.0 |  17M   56k
10-04 16:36:37|24.4 64.9  105| 19   0 0.8|78.2G    0  15.9G  326M|   0     0 | 
685k   21M|  39k 6163 | 94   2   4   0   0   1|87.5  57.8 |  19M   58k
10-04 16:36:47|24.1 63.5  104| 21   0 1.1|78.2G    0  15.9G  315M|   0     0 | 
632k   18M|  34k 5885 | 95   2   3   0   0   0|63.8  53.1 |  16M   65k
10-04 16:36:57|23.8 62.2  103| 21   0 0.9|78.2G    0  15.9G  310M|   0     0 | 
795k   19M|  34k 5634 | 95   2   2   0   0   0|75.7  53.4 |  16M   64k
10-04 16:37:07|24.1 61.0  102| 23   0 1.1|78.2G    0  15.9G  317M|   0     0 | 
869k   15M|  40k   13k| 94   2   4   0   0   1|91.6  47.8 |  16M  282k
10-04 16:37:17|24.6 59.9  101| 21   0 1.0|78.2G    0  15.9G  312M|   0     0 | 
730k   16M|  54k   30k| 93   2   5   0   0   1|81.6  48.3 |  14M  257k
10-04 16:37:27|23.4 58.4  100| 18   0 1.2|78.3G    0  15.8G  314M|   0     0 | 
651k   15M|  61k   40k| 89   3   7   0   0   1|74.3  47.1 |  17M  331k
10-04 16:37:37|24.4 57.5 99.5| 20   0 0.8|78.3G    0  15.8G  325M|   0     0 | 
782k   15M|  78k   76k| 87   4   8   0   0   1|57.6  41.8 |  13M 2531k
10-04 16:37:47|24.1 56.3 98.7| 21   0 1.0|78.3G    0  15.8G  308M|   0     0 
|1284k   18M|  67k   47k| 94   3   2   0   0   1| 128  58.6 |  19M 1835k
10-04 16:37:57|23.6 55.2 97.8| 21 0.1 1.0|78.3G    0  15.8G  318M|   0     0 
|1279k   19M|  40k 5963 | 96   2   2   0   0   0| 107  56.3 |  17M   73k
10-04 16:38:07|23.2 54.0 97.0| 21   0 0.8|78.3G    0  15.8G  301M|   0     0 
|1110k   18M|  38k 5986 | 96   2   2   0   0   0| 114  49.2 |  16M   70k
10-04 16:38:17|22.7 52.9 96.2| 20 0.1 0.8|78.3G    0  15.8G  321M|   0     0 
|1286k   21M|  39k 5773 | 96   2   1   0   0   0| 109  58.0 |  17M   46k
10-04 16:38:27|22.1 51.8 95.4| 18   0 1.5|78.3G    0  15.8G  314M|   0     0 
|2701k   21M|  50k 6534 | 91   2   5   0   0   1| 282  68.3 |  23M  246k
10-04 16:38:37|22.4 50.9 94.6| 20   0 0.8|78.3G    0  15.8G  324M|   0     0 
|1760k   17M|  40k 5498 | 94   2   3   0   0   1| 234  48.3 |  17M   64k
10-04 16:38:47|22.2 49.9 93.8| 21 0.1 0.9|78.3G    0  15.8G  311M|   0     0 
|1295k   18M|  42k 5610 | 95   2   3   0   0   0| 136  53.1 |  17M   55k
10-04 16:38:57|22.1 49.0 93.0| 21   0 0.8|78.3G    0  15.8G  316M|   0     0 
|1315k   19M|  44k 5387 | 96   2   2   0   0   0|97.4  55.1 |  18M   48k
10-04 16:39:07|18.9 47.4 92.1|2.0   0 0.8|78.3G    0  15.8G  308M|   0     0 | 
214k 2818k|7171  6043 | 20   0  79   0   0   0|13.8  7.80 |1691k 6620B
10-04 16:39:17|16.1 45.9 91.1|1.0   0 0.9|78.3G    0  15.8G  308M|   0     0 |  
16k 4864B|1263   200 |  6   0  94   0   0   0|0.50  0.60 |1912B  547B
10-04 16:39:27|13.8 44.4 90.1|1.0   0 0.8|78.3G    0  15.8G  308M|   0     0 |  
 0     0 |1226   166 |  6   0  94   0   0   0|   0     0 |1721B  515B
10-04 16:39:37|11.7 42.9 89.2|1.0   0 0.8|78.3G    0  15.8G  309M|   0     0 |  
 0   449k|1217   162 |  6   0  94   0   0   0|   0  1.80 |1701B  398B
{code}
I will try again nevertheless.

> OutOfMemoryError when bootstrapping with less than 100GB RAM
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-14239
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14239
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Details of the bootstrapping Node
>  * ProLiant BL460c G7
>  * 56GB RAM
>  * 2x 146GB 10K HDD (One dedicated for Commitlog, one for Data, Hints and 
> saved_caches)
>  * CentOS 7.4 on SD-Card
>  * /tmp and /var/log on tmpfs
>  * Oracle JDK 1.8.0_151
>  * Cassandra 3.11.1
> Cluster
>  * 10 existing Nodes (Up and Normal)
>            Reporter: Jürgen Albersdorfer
>            Priority: Major
>         Attachments: Objects-by-class.csv, 
> Objects-with-biggest-retained-size.csv, cassandra-env.sh, cassandra.yaml, 
> gc.log.0.current.zip, jvm.options, jvm_opts.txt, stack-traces.txt
>
>
> Hi, I face an issue when bootstrapping a Node having less than 100GB RAM on 
> our 10 Node C* 3.11.1 Cluster.
> During bootstrap, when I watch the cassandra.log I observe a growth in JVM 
> Heap Old Gen which gets not significantly freed up any more.
> I know that JVM collects on Old Gen only when really needed. I can see 
> collections, but there is always a remainder which seems to grow forever 
> without ever getting freed.
> After the Node successfully Joined the Cluster, I can remove the extra RAM I 
> have given it for bootstrapping without any further effect.
> It feels like Cassandra will not forget about every single byte streamed over 
> the Network over time during bootstrapping, - which would be a memory leak 
> and a major problem, too.
> I was able to produce a HeapDumpOnOutOfMemoryError from a 56GB Node (40 GB 
> assigned JVM Heap). YourKit Profiler shows huge amount of Memory allocated 
> for org.apache.cassandra.db.Memtable (22 GB) 
> org.apache.cassandra.db.rows.BufferCell (19 GB) and java.nio.HeapByteBuffer 
> (11 GB)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to