[
https://issues.apache.org/jira/browse/HDDS-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17495779#comment-17495779
]
Shawn commented on HDDS-6345:
-----------------------------
I think I found the issue. It is because the WAL file is unbounded, and it was
getting very large, which took lots of memory to flush. See below file size:
{code:java}
total 380767712
-rw-r--r-- 1 shawn staff 37 Jan 26 17:06 IDENTITY
-rw-r--r-- 1 shawn staff 1214 Feb 2 16:31 000383.sst
-rw-r--r-- 1 shawn staff 1380 Feb 2 16:31 000380.sst
-rw-r--r-- 1 shawn staff 1077 Feb 2 16:31 000379.sst
-rw-r--r-- 1 shawn staff 83278288 Feb 3 14:06 LOG.old.1644367180146868
-rw-r--r-- 1 shawn staff 191230 Feb 8 16:41 LOG.old.1644367862134017
-rw-r--r-- 1 shawn staff 566890 Feb 8 17:51 LOG.old.1644371650684465
-rw-r--r-- 1 shawn staff 304362 Feb 8 18:14 LOG.old.1644373822259296
-rw-r--r-- 1 shawn staff 505381 Feb 8 19:20 LOG.old.1644377425512645
-rw-r--r-- 1 shawn staff 5284023 Feb 8 20:54 LOG.old.1644382480517261
-rw-r--r-- 1 shawn staff 106279 Feb 8 20:56 LOG.old.1644382661478789
-rw-r--r-- 1 shawn staff 106467 Feb 8 20:59 LOG.old.1644382901675042
-rw-r--r-- 1 shawn staff 106556 Feb 8 21:04 LOG.old.1644383138686925
-rw-r--r-- 1 shawn staff 106645 Feb 8 21:08 LOG.old.1644383379912812
-rw-r--r-- 1 shawn staff 106734 Feb 8 21:12 LOG.old.1644383620030288
-rw-r--r-- 1 shawn staff 106823 Feb 8 21:16 LOG.old.1644384162299772
-rw-r--r-- 1 shawn staff 106912 Feb 8 21:25 LOG.old.1644384398832502
-rw-r--r-- 1 shawn staff 107001 Feb 8 21:29 LOG.old.1644384950145359
-rw-r--r-- 1 shawn staff 107090 Feb 8 21:38 LOG.old.1644385179106091
-rw-r--r-- 1 shawn staff 107179 Feb 8 21:42 LOG.old.1644385726720152
-rw-r--r-- 1 shawn staff 107268 Feb 8 21:51 LOG.old.1644385958622999
-rw-r--r-- 1 shawn staff 107357 Feb 8 21:55 LOG.old.1644386502402807
-rw-r--r-- 1 shawn staff 107446 Feb 8 22:04 LOG.old.1644386738532510
-rw-r--r-- 1 shawn staff 107535 Feb 8 22:08 LOG.old.1644387288327346
-rw-r--r-- 1 shawn staff 107624 Feb 8 22:17 LOG.old.1644387518807277
-rw-r--r-- 1 shawn staff 107713 Feb 8 22:21 LOG.old.1644388062294662
-rw-r--r-- 1 shawn staff 107802 Feb 8 22:30 LOG.old.1644388300669103
-rw-r--r-- 1 shawn staff 107891 Feb 8 22:34 LOG.old.1644388846337014
-rw-r--r-- 1 shawn staff 107980 Feb 8 22:43 LOG.old.1644389078313938
-rw-r--r-- 1 shawn staff 108069 Feb 8 22:47 LOG.old.1644389625014527
-rw-r--r-- 1 shawn staff 108158 Feb 8 22:56 LOG.old.1644389861691957
-rw-r--r-- 1 shawn staff 108247 Feb 8 23:00 LOG.old.1644390400549194
-rw-r--r-- 1 shawn staff 108336 Feb 8 23:09 LOG.old.1644390640571235
-rw-r--r-- 1 shawn staff 108425 Feb 8 23:13 LOG.old.1644391180685478
-rw-r--r-- 1 shawn staff 108514 Feb 8 23:22 LOG.old.1644391420337996
-rw-r--r-- 1 shawn staff 108603 Feb 8 23:26 LOG.old.1644391964725211
-rw-r--r-- 1 shawn staff 108692 Feb 8 23:35 LOG.old.1644392199725390
-rw-r--r-- 1 shawn staff 108781 Feb 8 23:39 LOG.old.1644392740994847
-rw-r--r-- 1 shawn staff 108870 Feb 8 23:48 LOG.old.1644392979153757
-rw-r--r-- 1 shawn staff 108959 Feb 8 23:52 LOG.old.1644393524508946
-rw-r--r-- 1 shawn staff 109048 Feb 9 00:01 LOG.old.1644393758536120
-rw-r--r-- 1 shawn staff 109137 Feb 9 00:05 LOG.old.1644394301634888
-rw-r--r-- 1 shawn staff 109226 Feb 9 00:14 LOG.old.1644394540693239
-rw-r--r-- 1 shawn staff 109315 Feb 9 00:18 LOG.old.1644395086388527
-rw-r--r-- 1 shawn staff 109404 Feb 9 00:27 LOG.old.1644395320594951
-rw-r--r-- 1 shawn staff 109493 Feb 9 00:31 LOG.old.1644395867586278
-rw-r--r-- 1 shawn staff 109582 Feb 9 00:40 LOG.old.1644396099547501
-rw-r--r-- 1 shawn staff 109671 Feb 9 00:44 LOG.old.1644396639528847
-rw-r--r-- 1 shawn staff 109760 Feb 9 00:53 LOG.old.1644396880902533
-rw-r--r-- 1 shawn staff 109849 Feb 9 00:57 LOG.old.1644397427125903
-rw-r--r-- 1 shawn staff 109938 Feb 9 01:06 LOG.old.1644397659192914
-rw-r--r-- 1 shawn staff 110027 Feb 9 01:10 LOG.old.1644398033442045
-rw-r--r-- 1 shawn staff 110116 Feb 9 01:16 LOG.old.1644398258050254
-rw-r--r-- 1 shawn staff 110205 Feb 9 01:20 LOG.old.1644398499697716
-rw-r--r-- 1 shawn staff 110294 Feb 9 01:24 LOG.old.1644398738900193
-rw-r--r-- 1 shawn staff 110383 Feb 9 01:28 LOG.old.1644398979546640
-rw-r--r-- 1 shawn staff 110472 Feb 9 01:32 LOG.old.1644399219839332
-rw-r--r-- 1 shawn staff 110561 Feb 9 01:36 LOG.old.1644399762490871
-rw-r--r-- 1 shawn staff 110650 Feb 9 01:45 LOG.old.1644399998917090
-rw-r--r-- 1 shawn staff 110739 Feb 9 01:49 LOG.old.1644400544520294
-rw-r--r-- 1 shawn staff 110828 Feb 9 01:58 LOG.old.1644400778363907
-rw-r--r-- 1 shawn staff 110917 Feb 9 02:02 LOG.old.1644401321026469
-rw-r--r-- 1 shawn staff 111006 Feb 9 02:11 LOG.old.1644401559481081
-rw-r--r-- 1 shawn staff 111095 Feb 9 02:15 LOG.old.1644402101564471
-rw-r--r-- 1 shawn staff 111184 Feb 9 02:24 LOG.old.1644402338910606
-rw-r--r-- 1 shawn staff 111273 Feb 9 02:28 LOG.old.1644402886999920
-rw-r--r-- 1 shawn staff 111362 Feb 9 02:37 LOG.old.1644403119687721
-rw-r--r-- 1 shawn staff 111451 Feb 9 02:41 LOG.old.1644403667253644
-rw-r--r-- 1 shawn staff 111540 Feb 9 02:50 LOG.old.1644403899658573
-rw-r--r-- 1 shawn staff 111629 Feb 9 02:54 LOG.old.1644404445198618
-rw-r--r-- 1 shawn staff 111718 Feb 9 03:03 LOG.old.1644404680680295
-rw-r--r-- 1 shawn staff 111807 Feb 9 03:07 LOG.old.1644405226951079
-rw-r--r-- 1 shawn staff 111896 Feb 9 03:16 LOG.old.1644405460216014
-rw-r--r-- 1 shawn staff 111985 Feb 9 03:20 LOG.old.1644406006312878
-rw-r--r-- 1 shawn staff 112074 Feb 9 03:29 LOG.old.1644406238740403
-rw-r--r-- 1 shawn staff 112163 Feb 9 03:33 LOG.old.1644406784735813
-rw-r--r-- 1 shawn staff 112252 Feb 9 03:42 LOG.old.1644407018774890
-rw-r--r-- 1 shawn staff 112341 Feb 9 03:46 LOG.old.1644407562139133
-rw-r--r-- 1 shawn staff 112430 Feb 9 03:55 LOG.old.1644407799778179
-rw-r--r-- 1 shawn staff 112519 Feb 9 03:59 LOG.old.1644408342693418
-rw-r--r-- 1 shawn staff 112608 Feb 9 04:08 LOG.old.1644408580269484
-rw-r--r-- 1 shawn staff 112697 Feb 9 04:12 LOG.old.1644409126569830
-rw-r--r-- 1 shawn staff 112786 Feb 9 04:21 LOG.old.1644409361232784
-rw-r--r-- 1 shawn staff 112875 Feb 9 04:25 LOG.old.1644409902070652
-rw-r--r-- 1 shawn staff 112964 Feb 9 04:34 LOG.old.1644410140032951
-rw-r--r-- 1 shawn staff 113053 Feb 9 04:38 LOG.old.1644410690786336
-rw-r--r-- 1 shawn staff 113142 Feb 9 04:47 LOG.old.1644410919482007
-rw-r--r-- 1 shawn staff 113231 Feb 9 04:51 LOG.old.1644411470550974
-rw-r--r-- 1 shawn staff 113320 Feb 9 05:00 LOG.old.1644411699036985
-rw-r--r-- 1 shawn staff 113409 Feb 9 05:04 LOG.old.1644412241555517
-rw-r--r-- 1 shawn staff 113498 Feb 9 05:13 LOG.old.1644412479480455
-rw-r--r-- 1 shawn staff 113587 Feb 9 05:17 LOG.old.1644413031432940
-rw-r--r-- 1 shawn staff 113676 Feb 9 05:26 LOG.old.1644413259744797
-rw-r--r-- 1 shawn staff 113765 Feb 9 05:30 LOG.old.1644413806660448
-rw-r--r-- 1 shawn staff 113854 Feb 9 05:39 LOG.old.1644414039589257
-rw-r--r-- 1 shawn staff 113943 Feb 9 05:43 LOG.old.1644414591808955
-rw-r--r-- 1 shawn staff 114032 Feb 9 05:52 LOG.old.1644414819345416
-rw-r--r-- 1 shawn staff 114121 Feb 9 05:56 LOG.old.1644415362906326
-rw-r--r-- 1 shawn staff 114210 Feb 9 06:05 LOG.old.1644415600199073
-rw-r--r-- 1 shawn staff 114299 Feb 9 06:09 LOG.old.1644416147359502
-rw-r--r-- 1 shawn staff 114388 Feb 9 06:18 LOG.old.1644416380002880
-rw-r--r-- 1 shawn staff 114477 Feb 9 06:22 LOG.old.1644416927426391
-rw-r--r-- 1 shawn staff 114566 Feb 9 06:31 LOG.old.1644417159154814
-rw-r--r-- 1 shawn staff 114655 Feb 9 06:35 LOG.old.1644417713480412
-rw-r--r-- 1 shawn staff 114744 Feb 9 06:44 LOG.old.1644417938617596
-rw-r--r-- 1 shawn staff 114833 Feb 9 06:48 LOG.old.1644418492949639
-rw-r--r-- 1 shawn staff 114922 Feb 9 06:57 LOG.old.1644418719162249
-rw-r--r-- 1 shawn staff 115011 Feb 9 07:01 LOG.old.1644419269448318
-rw-r--r-- 1 shawn staff 115100 Feb 9 07:10 LOG.old.1644419499649704
-rw-r--r-- 1 shawn staff 115189 Feb 9 07:14 LOG.old.1644420049951957
-rw-r--r-- 1 shawn staff 115278 Feb 9 07:23 LOG.old.1644420280145581
-rw-r--r-- 1 shawn staff 115367 Feb 9 07:27 LOG.old.1644420650942114
-rw-r--r-- 1 shawn staff 115456 Feb 9 07:33 LOG.old.1644420879120750
-rw-r--r-- 1 shawn staff 115545 Feb 9 07:37 LOG.old.1644421119364702
-rw-r--r-- 1 shawn staff 115634 Feb 9 07:41 LOG.old.1644421360384999
-rw-r--r-- 1 shawn staff 115723 Feb 9 07:45 LOG.old.1644421599500920
-rw-r--r-- 1 shawn staff 115812 Feb 9 07:49 LOG.old.1644421840309050
-rw-r--r-- 1 shawn staff 115901 Feb 9 07:53 LOG.old.1644422383315951
-rw-r--r-- 1 shawn staff 115990 Feb 9 08:02 LOG.old.1644422619523720
-rw-r--r-- 1 shawn staff 116079 Feb 9 08:06 LOG.old.1644423159398227
-rw-r--r-- 1 shawn staff 116168 Feb 9 08:15 LOG.old.1644423399376706
-rw-r--r-- 1 shawn staff 116257 Feb 9 08:19 LOG.old.1644423943384291
-rw-r--r-- 1 shawn staff 116346 Feb 9 08:28 LOG.old.1644424178622936
-rw-r--r-- 1 shawn staff 116435 Feb 9 08:32 LOG.old.1644424732167096
-rw-r--r-- 1 shawn staff 116524 Feb 9 08:41 LOG.old.1644424958806097
-rw-r--r-- 1 shawn staff 116613 Feb 9 08:45 LOG.old.1644425509010589
-rw-r--r-- 1 shawn staff 116702 Feb 9 08:54 LOG.old.1644425739325300
-rw-r--r-- 1 shawn staff 116791 Feb 9 08:58 LOG.old.1644426282553924
-rw-r--r-- 1 shawn staff 116880 Feb 9 09:07 LOG.old.1644426521105547
-rw-r--r-- 1 shawn staff 116969 Feb 9 09:11 LOG.old.1644427067819954
-rw-r--r-- 1 shawn staff 117058 Feb 9 09:20 LOG.old.1644427299554273
-rw-r--r-- 1 shawn staff 117147 Feb 9 09:24 LOG.old.1644427841637937
-rw-r--r-- 1 shawn staff 117236 Feb 9 09:33 LOG.old.1644428079255698
-rw-r--r-- 1 shawn staff 117325 Feb 9 09:37 LOG.old.1644428662441875
-rw-r--r-- 1 shawn staff 116958 Feb 9 09:44 LOG.old.1644429847119288
-rw-r--r-- 1 shawn staff 117503 Feb 9 10:06 LOG.old.1644430148899049
-rw-r--r-- 1 shawn staff 117592 Feb 9 10:11 LOG.old.1644430409502934
-rw-r--r-- 1 shawn staff 117681 Feb 9 10:16 LOG.old.1644430702032723
-rw-r--r-- 1 shawn staff 117770 Feb 9 10:21 LOG.old.1644431048047149
-rw-r--r-- 1 shawn staff 117760 Feb 9 10:25 LOG.old.1644431189591258
-rw-r--r-- 1 shawn staff 117948 Feb 9 10:28 LOG.old.1644431597998732
-rw-r--r-- 1 shawn staff 118037 Feb 9 10:35 LOG.old.1644432210087151
-rw-r--r-- 1 shawn staff 118126 Feb 9 10:46 LOG.old.1644432448733965
-rw-r--r-- 1 shawn staff 118215 Feb 9 10:50 LOG.old.1644432741009948
-rw-r--r-- 1 shawn staff 118304 Feb 9 10:55 LOG.old.1644433095650442
-rw-r--r-- 1 shawn staff 118294 Feb 9 10:59 LOG.old.1644433228372320
-rw-r--r-- 1 shawn staff 118482 Feb 9 11:02 LOG.old.1644433644867623
-rw-r--r-- 1 shawn staff 118571 Feb 9 11:10 LOG.old.1644433947759091
-rw-r--r-- 1 shawn staff 118660 Feb 9 11:15 LOG.old.1644434554154131
-rw-r--r-- 1 shawn staff 118749 Feb 9 11:25 LOG.old.1644434853588250
-rw-r--r-- 1 shawn staff 118838 Feb 9 11:30 LOG.old.1644435181713238
-rw-r--r-- 1 shawn staff 118927 Feb 9 11:35 LOG.old.1644435449831331
-rw-r--r-- 1 shawn staff 118917 Feb 9 11:38 LOG.old.1644436611170224
-rw-r--r-- 1 shawn staff 1069587 Feb 9 14:11 LOG.old.1644464957066692
-rw-r--r-- 1 shawn staff 12189284 Feb 10 11:05 LOG.old.1644520174314072
-rw-r--r-- 1 shawn staff 106651 Feb 10 11:16 LOG.old.1644520638312149
-rw-r--r-- 1 shawn staff 32925837 Feb 10 16:41 LOG.old.1644541486786840
-rw-r--r-- 1 shawn staff 12376786 Feb 10 20:00 LOG.old.1644552344704739
-rw-r--r-- 1 shawn staff 1414 Feb 12 00:09 117619.sst
-rw-r--r-- 1 shawn staff 247240425 Feb 12 13:57 LOG.old.1644703381565528
-rw-r--r-- 1 shawn staff 4384424 Feb 12 18:19 LOG.old.1644719040579515
-rw-r--r-- 1 shawn staff 976635 Feb 12 20:17 LOG.old.1644726639987136
-rw-r--r-- 1 shawn staff 390557 Feb 12 21:00 LOG.old.1644729047763506
-rw-r--r-- 1 shawn staff 369416 Feb 12 21:40 LOG.old.1644731628208969
-rw-r--r-- 1 shawn staff 238152 Feb 12 22:03 LOG.old.1644732526638222
-rw-r--r-- 1 shawn staff 303784 Feb 12 22:28 LOG.old.1644734098159920
-rw-r--r-- 1 shawn staff 6801352 Feb 13 15:24 LOG.old.1644795387928568
-rw-r--r-- 1 shawn staff 566312 Feb 13 16:36 LOG.old.1644800989160276
-rw-r--r-- 1 shawn staff 303784 Feb 13 17:29 LOG.old.1644802806662781
-rw-r--r-- 1 shawn staff 1254713 Feb 13 19:04 LOG.old.1644808089666726
-rw-r--r-- 1 shawn staff 302105 Feb 13 19:27 LOG.old.1644809537175955
-rw-r--r-- 1 shawn staff 83697940 Feb 14 10:48 LOG.old.1644865165363296
-rw-r--r-- 1 shawn staff 182834 Feb 14 11:04 LOG.old.1644867077089980
-rw-r--r-- 1 shawn staff 172520 Feb 14 11:31 LOG.old.1644867416542094
-rw-r--r-- 1 shawn staff 369416 Feb 14 12:06 LOG.old.1644869808070612
-rw-r--r-- 1 shawn staff 303784 Feb 14 12:36 LOG.old.1644871386406761
-rw-r--r-- 1 shawn staff 172520 Feb 14 12:43 LOG.old.1644872153169418
-rw-r--r-- 1 shawn staff 2506059 Feb 14 13:35 LOG.old.1644874928951884
-rw-r--r-- 1 shawn staff 2619962 Feb 14 14:34 LOG.old.1644878264849453
-rw-r--r-- 1 shawn staff 4605960 Feb 14 15:33 LOG.old.1644881923568923
-rw-r--r-- 1 shawn staff 963151 Feb 14 16:05 LOG.old.1644884082853255
-rw-r--r-- 1 shawn staff 3914281 Feb 14 16:44 LOG.old.1644886223028686
-rw-r--r-- 1 shawn staff 9029720 Feb 14 18:15 LOG.old.1644891584994081
-rw-r--r-- 1 shawn staff 325427 Feb 14 18:32 LOG.old.1644894016729761
-rw-r--r-- 1 shawn staff 844833 Feb 14 20:40 LOG.old.1644900812851844
-rw-r--r-- 1 shawn staff 303784 Feb 14 21:13 LOG.old.1644902806010207
-rw-r--r-- 1 shawn staff 319270 Feb 14 21:42 LOG.old.1644904011794136
-rw-r--r-- 1 shawn staff 1447 Feb 15 08:10 196426.sst
-rw-r--r-- 1 shawn staff 881578 Feb 15 08:10 196424.sst
-rw-r--r-- 1 shawn staff 56689293 Feb 15 08:37 LOG.old.1644943174521190
-rw-r--r-- 1 shawn staff 3835 Feb 15 08:58 196465.sst
-rw-r--r-- 1 shawn staff 291264 Feb 15 09:08 LOG.old.1644944958636030
-rw-r--r-- 1 shawn staff 106946 Feb 15 09:11 LOG.old.1644946026884177
-rw-r--r-- 1 shawn staff 107035 Feb 15 09:28 LOG.old.1644947130284959
-rw-r--r-- 1 shawn staff 107124 Feb 15 09:47 LOG.old.1644948389780617
-rw-r--r-- 1 shawn staff 107213 Feb 15 10:08 LOG.old.1644949409639116
-rw-r--r-- 1 shawn staff 107302 Feb 15 10:25 LOG.old.1644950444253316
-rw-r--r-- 1 shawn staff 107391 Feb 15 10:42 LOG.old.1644951477750358
-rw-r--r-- 1 shawn staff 107480 Feb 15 10:59 LOG.old.1644952546409840
-rw-r--r-- 1 shawn staff 107569 Feb 15 11:17 LOG.old.1644953590670435
-rw-r--r-- 1 shawn staff 107658 Feb 15 11:35 LOG.old.1644955422424048
-rw-r--r-- 1 shawn staff 107747 Feb 15 12:05 LOG.old.1644956518350200
-rw-r--r-- 1 shawn staff 107836 Feb 15 12:23 LOG.old.1644957565999479
-rw-r--r-- 1 shawn staff 107925 Feb 15 12:41 LOG.old.1644958610859156
-rw-r--r-- 1 shawn staff 107913 Feb 15 12:57 LOG.old.1644958828039333
-rw-r--r-- 1 shawn staff 108103 Feb 15 13:02 LOG.old.1644959907194544
-rw-r--r-- 1 shawn staff 108192 Feb 15 13:20 LOG.old.1644960970628484
-rw-r--r-- 1 shawn staff 108281 Feb 15 13:37 LOG.old.1644962044184684
-rw-r--r-- 1 shawn staff 108370 Feb 15 13:55 LOG.old.1644963099602041
-rw-r--r-- 1 shawn staff 108459 Feb 15 14:13 LOG.old.1644964150661289
-rw-r--r-- 1 shawn staff 108548 Feb 15 14:31 LOG.old.1644965205900854
-rw-r--r-- 1 shawn staff 108637 Feb 15 14:48 LOG.old.1644966265969017
-rw-r--r-- 1 shawn staff 108726 Feb 15 15:06 LOG.old.1644967315330015
-rw-r--r-- 1 shawn staff 108815 Feb 15 15:23 LOG.old.1644968366498300
-rw-r--r-- 1 shawn staff 108904 Feb 15 15:41 LOG.old.1644969420530820
-rw-r--r-- 1 shawn staff 108993 Feb 15 15:58 LOG.old.1644970473861801
-rw-r--r-- 1 shawn staff 109082 Feb 15 16:16 LOG.old.1644971538699848
-rw-r--r-- 1 shawn staff 109171 Feb 15 16:34 LOG.old.1644972375777307
-rw-r--r-- 1 shawn staff 109260 Feb 15 16:48 LOG.old.1644973485657518
-rw-r--r-- 1 shawn staff 109349 Feb 15 17:06 LOG.old.1644973955229088
-rw-r--r-- 1 shawn staff 109438 Feb 15 17:14 LOG.old.1644975014885893
-rw-r--r-- 1 shawn staff 109527 Feb 15 17:32 LOG.old.1644975375017143
-rw-r--r-- 1 shawn staff 11514 Feb 15 17:36 LOG.old.1644975541732263
-rw-r--r-- 1 shawn staff 0 Feb 15 17:39 LOCK
-rw-r--r-- 1 shawn staff 97673 Feb 15 17:39 LOG.old.1644975938156564
-rw-r--r-- 1 shawn staff 109515 Feb 15 17:45 LOG.old.1644976204474304
-rw-r--r-- 1 shawn staff 97762 Feb 15 17:51 LOG.old.1644979134738960
-rw-r--r-- 1 shawn staff 97664 Feb 15 18:39 LOG.old.1644979176824228
-rw-r--r-- 1 shawn staff 186990 Feb 15 18:57 LOG.old.1644983399091891
-rw-r--r-- 1 shawn staff 890791 Feb 15 19:56 196832.sst
-rw-r--r-- 1 shawn staff 806968 Feb 15 20:05 LOG.old.1644984435743603
-rw-r--r-- 1 shawn staff 249576 Feb 15 20:14 LOG.old.1644985070476293
-rw-r--r-- 1 shawn staff 106150 Feb 15 20:19 LOG.old.1644985692475449
-rw-r--r-- 1 shawn staff 94216 Feb 15 20:28 LOG.old.1644985737066891
-rw-r--r-- 1 shawn staff 165750 Feb 15 20:35 LOG.old.1644986411873152
-rw-r--r-- 1 shawn staff 544810 Feb 15 21:00 LOG.old.1644987849854150
-rw-r--r-- 1 shawn staff 2907254 Feb 15 21:22 197033.sst
-rw-r--r-- 1 shawn staff 1101 Feb 15 21:22 197032.sst
-rw-r--r-- 1 shawn staff 1411 Feb 15 21:22 197036.sst
-rw-r--r-- 1 shawn staff 395259 Feb 15 21:32 LOG.old.1644990074886664
-rw-r--r-- 1 shawn staff 1386 Feb 15 21:46 197095.sst
-rw-r--r-- 1 shawn staff 1528 Feb 15 21:46 197093.sst
-rw-r--r-- 1 shawn staff 153783 Feb 15 21:46 197092.sst
-rw-r--r-- 1 shawn staff 1078 Feb 15 21:46 197091.sst
-rw-r--r-- 1 shawn staff 63828 Feb 15 21:47 OPTIONS-197099
-rw-r--r-- 1 shawn staff 63328073 Feb 15 21:47 197100.sst
-rw-r--r-- 1 shawn staff 370185 Feb 15 22:07 LOG.old.1644991832932491
-rw-r--r-- 1 shawn staff 1388 Feb 15 22:27 197131.sst
-rw-r--r-- 1 shawn staff 167612 Feb 15 22:27 197130.sst
-rw-r--r-- 1 shawn staff 3149 Feb 15 22:27 197129.sst
-rw-r--r-- 1 shawn staff 437691 Feb 15 22:27 197128.sst
-rw-r--r-- 1 shawn staff 1078 Feb 15 22:27 197127.sst
-rw-r--r-- 1 shawn staff 63828 Feb 15 22:28 OPTIONS-197135
-rw-r--r-- 1 shawn staff 123416865 Feb 15 22:28 197133.log
-rw-r--r-- 1 shawn staff 128261761 Feb 15 22:28 197136.log
-rw-r--r-- 1 shawn staff 125265074 Feb 15 22:28 197139.log
-rw-r--r-- 1 shawn staff 130482170 Feb 15 22:28 197143.log
-rw-r--r-- 1 shawn staff 137692148 Feb 15 22:28 197141.log
-rw-r--r-- 1 shawn staff 126780973 Feb 15 22:28 197146.log
-rw-r--r-- 1 shawn staff 145666726 Feb 15 22:28 197149.log
-rw-r--r-- 1 shawn staff 132462639 Feb 15 22:28 197151.log
-rw-r--r-- 1 shawn staff 126716615 Feb 15 22:28 197153.log
-rw-r--r-- 1 shawn staff 140423892 Feb 15 22:28 197156.log
-rw-r--r-- 1 shawn staff 150750575 Feb 15 22:28 197159.log
-rw-r--r-- 1 shawn staff 135953210 Feb 15 22:28 197161.log
-rw-r--r-- 1 shawn staff 133847252 Feb 15 22:28 197163.log
-rw-r--r-- 1 shawn staff 135592577 Feb 15 22:28 197166.log
-rw-r--r-- 1 shawn staff 131710862 Feb 15 22:28 197169.log
-rw-r--r-- 1 shawn staff 140533803 Feb 15 22:28 197171.log
-rw-r--r-- 1 shawn staff 150168673 Feb 15 22:28 197173.log
-rw-r--r-- 1 shawn staff 5615167 Feb 15 22:28 197179.sst
-rw-r--r-- 1 shawn staff 67461542 Feb 15 22:28 197178.sst
-rw-r--r-- 1 shawn staff 17356425 Feb 15 22:28 197181.sst
-rw-r--r-- 1 shawn staff 271432123 Feb 15 22:28 197176.log
-rw-r--r-- 1 shawn staff 17470070 Feb 15 22:31 197183.sst
-rw-r--r-- 1 shawn staff 2463909965 Feb 15 22:32 197180.log
-rw-r--r-- 1 shawn staff 21975 Feb 15 22:47 MANIFEST-197132
-rw-r--r-- 1 shawn staff 17716232 Feb 15 22:47 197185.sst
-rw-r--r-- 1 shawn staff 28311755951 Feb 15 22:47 197182.log
-rw-r--r-- 1 shawn staff 422059 Feb 15 22:48 LOG.old.1644994533640130
-rw-r--r-- 1 shawn staff 160802137543 Feb 15 22:52 197184.log
-rw-r--r-- 1 shawn staff 8370 Feb 15 22:55 MANIFEST-197188
-rw-r--r-- 1 shawn staff 108605 Feb 15 23:02 LOG.old.1644996671297950
-rw-r--r-- 1 shawn staff 8370 Feb 15 23:31 MANIFEST-197191
-rw-r--r-- 1 shawn staff 108694 Feb 15 23:37 LOG.old.1644998780436977
-rw-r--r-- 1 shawn staff 8370 Feb 16 00:06 MANIFEST-197194
-rw-r--r-- 1 shawn staff 108783 Feb 16 00:12 LOG.old.1645000885635251
-rw-r--r-- 1 shawn staff 8370 Feb 16 00:41 MANIFEST-197197
-rw-r--r-- 1 shawn staff 108872 Feb 16 00:47 LOG.old.1645003028944254
-rw-r--r-- 1 shawn staff 8370 Feb 16 01:17 MANIFEST-197200
-rw-r--r-- 1 shawn staff 108961 Feb 16 01:23 LOG.old.1645005152832911
-rw-r--r-- 1 shawn staff 8370 Feb 16 01:52 MANIFEST-197203
-rw-r--r-- 1 shawn staff 109050 Feb 16 01:58 LOG.old.1645007262008713
-rw-r--r-- 1 shawn staff 8370 Feb 16 02:27 MANIFEST-197206
-rw-r--r-- 1 shawn staff 109139 Feb 16 02:34 LOG.old.1645009386965249
-rw-r--r-- 1 shawn staff 8370 Feb 16 03:03 MANIFEST-197209
-rw-r--r-- 1 shawn staff 109228 Feb 16 03:09 LOG.old.1645011493586081
-rw-r--r-- 1 shawn staff 8370 Feb 16 03:38 MANIFEST-197212
-rw-r--r-- 1 shawn staff 109317 Feb 16 03:44 LOG.old.1645013620453700
-rw-r--r-- 1 shawn staff 8370 Feb 16 04:13 MANIFEST-197215
-rw-r--r-- 1 shawn staff 109406 Feb 16 04:19 LOG.old.1645015738040257
-rw-r--r-- 1 shawn staff 8370 Feb 16 04:48 MANIFEST-197218
-rw-r--r-- 1 shawn staff 109495 Feb 16 04:55 LOG.old.1645017828714614
-rw-r--r-- 1 shawn staff 8370 Feb 16 05:23 MANIFEST-197221
-rw-r--r-- 1 shawn staff 109584 Feb 16 05:30 LOG.old.1645019955778176
-rw-r--r-- 1 shawn staff 8370 Feb 16 05:59 MANIFEST-197224
-rw-r--r-- 1 shawn staff 109673 Feb 16 06:05 LOG.old.1645022056638563
-rw-r--r-- 1 shawn staff 8370 Feb 16 06:34 MANIFEST-197227
-rw-r--r-- 1 shawn staff 109762 Feb 16 06:40 LOG.old.1645024188526328
-rw-r--r-- 1 shawn staff 8370 Feb 16 07:09 MANIFEST-197230
-rw-r--r-- 1 shawn staff 109851 Feb 16 07:15 LOG.old.1645026321379464
-rw-r--r-- 1 shawn staff 8370 Feb 16 07:45 MANIFEST-197233
-rw-r--r-- 1 shawn staff 109940 Feb 16 07:51 LOG.old.1645028412533159
-rw-r--r-- 1 shawn staff 8370 Feb 16 08:20 MANIFEST-197236
-rw-r--r-- 1 shawn staff 110029 Feb 16 08:26 LOG.old.1645030530819666
-rw-r--r-- 1 shawn staff 8370 Feb 16 08:55 MANIFEST-197239
-rw-r--r-- 1 shawn staff 110118 Feb 16 09:01 LOG.old.1645032611452367
-rw-r--r-- 1 shawn staff 8370 Feb 16 09:30 MANIFEST-197242
-rw-r--r-- 1 shawn staff 110207 Feb 16 09:36 LOG.old.1645034727350086
-rw-r--r-- 1 shawn staff 8370 Feb 16 10:05 MANIFEST-197245
-rw-r--r-- 1 shawn staff 110296 Feb 16 10:11 LOG.old.1645036855733875
-rw-r--r-- 1 shawn staff 8370 Feb 16 10:40 MANIFEST-197248
-rw-r--r-- 1 shawn staff 110385 Feb 16 10:47 LOG.old.1645038973943606
-rw-r--r-- 1 shawn staff 8370 Feb 16 11:16 MANIFEST-197251
-rw-r--r-- 1 shawn staff 110474 Feb 16 11:22 LOG.old.1645041024285828
-rw-r--r-- 1 shawn staff 8370 Feb 16 11:50 MANIFEST-197254
-rw-r--r-- 1 shawn staff 110462 Feb 16 11:51 LOG.old.1645041761042691
-rw-r--r-- 1 shawn staff 97181 Feb 16 12:02 LOG.old.1645061592168614
-rw-r--r-- 1 shawn staff 8370 Feb 16 17:33 MANIFEST-197257
-rw-r--r-- 1 shawn staff 110717 Feb 16 17:33 LOG.old.1645121679178958
-rw-r--r-- 1 shawn staff 8370 Feb 17 10:14 MANIFEST-197260
-rw-r--r-- 1 shawn staff 110806 Feb 17 10:14 LOG.old.1645135938368839
-rw-r--r-- 1 shawn staff 8370 Feb 17 14:12 MANIFEST-197263
-rw-r--r-- 1 shawn staff 16 Feb 17 14:12 CURRENT
-rw-r--r-- 1 shawn staff 110996 Feb 17 14:15 LOG {code}
> OM always runs OOM in Kubernetes
> ---------------------------------
>
> Key: HDDS-6345
> URL: https://issues.apache.org/jira/browse/HDDS-6345
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Shawn
> Priority: Major
>
> I deployed ozone 1.21 to kubernetes with security enabled and with OM HA and
> SCM HA. However, one of the OM always gets restarted by Kubernetes because of
> OOM. Even I assigned 300GB memory, the OM still keeps restarting for OOM.
>
> After analysis, we found the OOM was because of rocksDB. When OM gets
> restarted, it first tries to open rocksDB. And during this time, rocksDB
> tries to do compaction, which eventually got OOM. So there are three question:
>
> 1. Why the OM got into this status?
> 2. Why rocksDB needs so much memory to do the compaction?
> 3. How to resolve this?
> Some info maybe useful for you. We directly deploy OM HA, not migrate from
> one OM to HA OM. The OM that has issues is a follower, not a leader. The
> underlying PVC we are using is SSD. Our traffic is mostly large objects, with
> size of hundreds GBs.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
