wangning has posted comments on this change. ( http://gerrit.cloudera.org:8080/15830 )
Change subject: [fs] fixed file num low live ratio metadata compaction policy when startup ...................................................................... Patch Set 3: (4 comments) http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG@18 PS2, Line 18: nsumption > nit: consumption Done http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG@20 PS2, Line 20: ion of live > nit: remove. Done http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG@24 PS2, Line 24: during startup. > Can you explain more about how this flag (which is to limit the number of m For sure it would be very good to make those files compacted. But it is no possible for nodes without sufficient memory to startup. If sometimes it killed by OOM killer , it may keeps fail forever without intervention of Ops. The metadata files inflate over physical memory is very common to us, I am in private deployment company which offer service to other company. The management right doesn't belong to us until some module failed. Even if we monitored the bad situation of compaction of kudu tserver, it is not possible to restart and adjust some parameter without customer's permission. How the flag could help avoiding OOM during startup is attached in CL. We some times cut in half of num of metadata files to start once, after some compactions and remove the flag and restart it again. With 1.5 times expectation of start time but could help the node restart even in not very good hardware. http://gerrit.cloudera.org:8080/#/c/15830/2/src/kudu/fs/log_block_manager.cc File src/kudu/fs/log_block_manager.cc: http://gerrit.cloudera.org:8080/#/c/15830/2/src/kudu/fs/log_block_manager.cc@2631 PS2, Line 2631: FLAGS_max_num_low_live_ratio_metadata_file_to_compact > : l > nit: alignment. Done -- To view, visit http://gerrit.cloudera.org:8080/15830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896 Gerrit-Change-Number: 15830 Gerrit-PatchSet: 3 Gerrit-Owner: wangning <[email protected]> Gerrit-Reviewer: Hao Hao <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: wangning <[email protected]> Gerrit-Comment-Date: Wed, 29 Apr 2020 00:59:26 +0000 Gerrit-HasComments: Yes
