Hello Kudu Jenkins, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15830
to look at the new patch set (#4).
Change subject: [fs] fixed file num low live ratio metadata compaction policy
when startup
......................................................................
[fs] fixed file num low live ratio metadata compaction policy when
startup
During tserver startup, it loads all low live ratio metadata into memory,
and this is not under controll of --memory_limit_hard_bytes.
To those tservers which didn't compact well due to starvation of
compaction, the metadata files inflated, num of metadata files increases
and may result in OOM in next time startup. Sometimes the memory
consumption for startup may excess physical memory.
To limit the memory consumption of livemetadata compaction, we always
use --log_container_live_metadata_before_compact_ratio. But it is
not flexible cause we have no idea about the distribution of live ratio
of blocks.
Limit the num of metadata files to be compacted could help avoid OOM
during startup.
E.g. In a node with 32G physical memory, we found about 90GiB metatatada
files, the num of metadata files is 200k. It would be very good to
compact them all when startup. But it could take about 90GiB * 60% =
54GiB memory when starting(Not precise, as we observed is consume 60% of
metadata bytes in disk), obviously it excess the physical memory. So it keeps
OOM and cdh-agent keeps restart it. By passing the
--max_num_low_live_ratio_metadata_file_to_compact to limit the num of
low live ratio files could be selected to be compact, could make it
success to startup.
Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
2 files changed, 14 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/15830/4
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 4
Gerrit-Owner: wangning <[email protected]>
Gerrit-Reviewer: Hao Hao <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: wangning <[email protected]>
