[
https://issues.apache.org/jira/browse/KUDU-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713282#comment-17713282
]
Alexey Serbin edited comment on KUDU-3463 at 4/17/23 8:57 PM:
--------------------------------------------------------------
[~weizisheng],
Just two more cents in addition to what [~zhangyifan27] already pointed.
Even without deleting/creating tables in the system catalog, {{kudu-master}}
process might consume a lot of memory when receiving many updates on the tablet
status and DDLs like adding/dropping/changing columns in a table. See
[KUDU-3406|https://issues.apache.org/jira/browse/KUDU-3406] and
[KUDU-3429|https://issues.apache.org/jira/browse/KUDU-3429] for details. The
issue manifests itself in spiking memory usage for {{kudu-master}} when running
{{CompactRowSetsOp}}: you could use that as a criterion to verify whether the
issue that you see is related to
[KUDU-3406|https://issues.apache.org/jira/browse/KUDU-3406].
You can easily address that by setting {{\-\-tablet_history_max_age_sec=900}}
for kudu-masters. In addition, consider picking up
[4537be98d|https://github.com/apache/kudu/commit/4537be98d0542d99ecbe4674197b9d894c3eb4e5]
if you don't want to override the {{\-\-tablet_history_max_age_sec}} for
{{kudu-master}} processes.
If that's indeed the root cause of the issue, for kudu-master processes this
will be addressed in upcoming 1.17.0 release because it contains
[4537be98d|https://github.com/apache/kudu/commit/4537be98d0542d99ecbe4674197b9d894c3eb4e5].
Thanks!
was (Author: aserbin):
[~weizisheng],
Just two more cents in addition to what [~zhangyifan27] already pointed.
Even without deleting/creating tables in the system catalog, {{kudu-master}}
process might consume a lot of memory when receiving many updates on the tablet
status and DDLs like adding/dropping/changing columns in a table. See
[KUDU-3406|https://issues.apache.org/jira/browse/KUDU-3406] and
[KUDU-3429|https://issues.apache.org/jira/browse/KUDU-3429] for details.
You can easily address that by setting {{\-\-tablet_history_max_age_sec=900}}
for kudu-masters. In addition, consider picking up
[4537be98d|https://github.com/apache/kudu/commit/4537be98d0542d99ecbe4674197b9d894c3eb4e5]
if you don't want to override the {{\-\-tablet_history_max_age_sec}} for
{{kudu-master}} processes.
> KuduMaster leader consumes too much memory
> ------------------------------------------
>
> Key: KUDU-3463
> URL: https://issues.apache.org/jira/browse/KUDU-3463
> Project: Kudu
> Issue Type: Bug
> Affects Versions: 1.12.0
> Reporter: Weizisheng
> Priority: Major
> Attachments: heap321.txt
>
>
> We rarely face a suspected memory leak on a cluster with 3-master and
> 4-tserver, 800 tables and 3000 tablets. Leader master consume 50GB memory
> while the other two only take one tenth of it ...
> From the last 5 days, leader's memory usage grows 3%+
> attachment pprof heap
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
