[
https://issues.apache.org/jira/browse/KUDU-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martonka Zoltan updated KUDU-3812:
----------------------------------
Description:
If we just start a normal Kudu cluster and do the following:
+ Increase cell limit to 1MB.
+ Increase rpc_max_message_size, so reads does not fail with it.
(also tablet_transaction_memory_limit_mb, so flag validator does not
fail).
+ Create a table and start to put serious read and write load on it.
It will quickly crash with OOM issues. It has 3 identified problems:
+ Writes and heartbeats take way too much memory if the write buffer size
is increased. This is not really a problem, because we can keep the
default in 7 GB. And just send few rows at once.
+ Compaction takes up way too much memory due to fixed row count.
+ Ordered scans can take up a whole GB, that is not counted towards memory
tracking.
If memory is already kept near hard limit (80% of system memory by default)
by compaction or flushing, then this can easily cause a 8-10GB load with
the default 10 rpc. Which means we can theoretically crash kudu on proper
server hardware too (e.g 64 GB memory), not just the bare minimum
recommended hardware.
> Fix ordered scan OOM crash in case of 1MB cells.
> ------------------------------------------------
>
> Key: KUDU-3812
> URL: https://issues.apache.org/jira/browse/KUDU-3812
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Martonka Zoltan
> Priority: Major
>
> If we just start a normal Kudu cluster and do the following:
> + Increase cell limit to 1MB.
> + Increase rpc_max_message_size, so reads does not fail with it.
> (also tablet_transaction_memory_limit_mb, so flag validator does not
> fail).
> + Create a table and start to put serious read and write load on it.
> It will quickly crash with OOM issues. It has 3 identified problems:
> + Writes and heartbeats take way too much memory if the write buffer size
> is increased. This is not really a problem, because we can keep the
> default in 7 GB. And just send few rows at once.
> + Compaction takes up way too much memory due to fixed row count.
> + Ordered scans can take up a whole GB, that is not counted towards memory
> tracking.
> If memory is already kept near hard limit (80% of system memory by default)
> by compaction or flushing, then this can easily cause a 8-10GB load with
> the default 10 rpc. Which means we can theoretically crash kudu on proper
> server hardware too (e.g 64 GB memory), not just the bare minimum
> recommended hardware.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
