[
https://issues.apache.org/jira/browse/KUDU-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132440#comment-18132440
]
ASF subversion and git services commented on KUDU-3812:
-------------------------------------------------------
Commit c357d074efc78718f2aaa2750808fc938838e6f9 in kudu's branch
refs/heads/master from Zoltan Martonka
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=c357d074e ]
KUDU-3812 Test ordered scan memory usage with large cells.
Currently sorted scans have a burned in 1024 size buffer.
With 1 MB of cells this can cause large, untracked memory usage.
This test does the following:
+ Start a cluster with 1 tserver and create a tablet.
+ Disable compaction for worst case result (compaction falls behind).
+ Write some rows with 1 MB cells into a tablet.
+ Disable flushing then keep writing until memory limit is hit.
+ Start 10 parallel ordered full scans. They are started one-by-one with a
gap between them, so they probably request other parts of the data at
the same time.
+ Check the peak memory usage.
The test increases RPC and transaction limits, but the whole scanning still
happens when they are set to default. So the memory overconsumption would still
happen.
One run gave me:
Peak tmalloc memory during test : 18697 MiB
We will need to dynamically downsize batches when cells are really large,
so the memory usage won't skyrocket and rpc/transaction limits can remain
untouched.
Change-Id: I75ceaebac1793c2421302f8b205ba85ad32a141a
Reviewed-on: http://gerrit.cloudera.org:8080/24731
Tested-by: Zoltan Martonka <[email protected]>
Reviewed-by: Alexey Serbin <[email protected]>
Reviewed-by: Gabriella Lotz <[email protected]>
> Fix ordered scan OOM crash in case of 1MB cells.
> ------------------------------------------------
>
> Key: KUDU-3812
> URL: https://issues.apache.org/jira/browse/KUDU-3812
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Martonka Zoltan
> Priority: Major
>
> If we just start a normal Kudu cluster and do the following:
> + Increase cell limit to 1MB.
> + Increase rpc_max_message_size, so reads does not fail with it.
> (also tablet_transaction_memory_limit_mb, so flag validator does not
> fail).
> + Create a table and start to put serious read and write load on it.
> It will quickly crash with OOM issues. It has 3 identified problems:
> + Writes and heartbeats take way too much memory if the write buffer size
> is increased. This is not really a problem, because we can keep the
> default in 7 GB. And just send few rows at once.
> + Compaction takes up way too much memory due to fixed row count.
> + Ordered scans can take up a whole GB, that is not counted towards memory
> tracking.
> If memory is already kept near hard limit (80% of system memory by default)
> by compaction or flushing, then this can easily cause a 8-10GB load with
> the default 10 rpc. Which means we can theoretically crash kudu on proper
> server hardware too (e.g 64 GB memory), not just the bare minimum
> recommended hardware.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)