[
https://issues.apache.org/jira/browse/KUDU-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272545#comment-16272545
]
ZhangZhen commented on KUDU-2226:
---------------------------------
The table has almost 15K DRSs in one tablet, which results in 15K DMS in
memory, every DMS consume 60KB memory, including data and overhead, that's
900MB memory, thanks to [~jdcryans] for helping me figure this out.
According to JD's suggestion, I add more maintenance thread(default 1, set to
5), and set maintenance_manager_polling_interval_ms to 50 instead of default
250, hope the 15K DRSs can got a change to be compacted. After restarting the
server, I can see more maintenance instances running, but I find most of the
operations finished are "FlushMRSOp", "UndoDeltaBlockGCOp" etc, it seems the
CompactRowSetsOp get a low maintenance score and will not be executed.
> Frequently updated table does not flush DeltaMemStore in time and will occupy
> a lot of memory
> ---------------------------------------------------------------------------------------------
>
> Key: KUDU-2226
> URL: https://issues.apache.org/jira/browse/KUDU-2226
> Project: Kudu
> Issue Type: Improvement
> Affects Versions: 1.3.0
> Environment: CentOS6.5 Linux 2.6.32-431
> Kudu1.3.0
> GitCommit 00813f96b9cb
> Reporter: ZhangZhen
>
> I have a table with 10M rows in total and has been hash partitioned to 16
> buckets. Each tablet has about 100MB on disk size according to the /tablets
> Web UI. Everyday 50K new rows will be inserted into this table, and about 5M
> rows of this table will be updated, that's about half of rows in total, each
> row will be updated only once.
> Then I found something strange, from the /mem-trackers UI of TS, I found
> every tablet of this table occupied about 900MB memory, mainly occupied by
> DeltaMemStore, the peak memory consumption is about 1.8G.
> I don't understand why the DeltaMemStore will cost so much memory, 900MB DMS
> vs 100MB on disk size, that seems strange to me. What's more, I found these
> DMS will be flushed very slowly, so for a long time these memory are
> occupied, which cause "Soft memory limit exceeded" in the TS, and in result
> cause "Rejecting consensus request".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
