[
https://issues.apache.org/jira/browse/KUDU-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146883#comment-18146883
]
ASF subversion and git services commented on KUDU-3812:
-------------------------------------------------------
Commit 74304f5f243a6e002551a4dcb53cc3941ca77e02 in kudu's branch
refs/heads/master from Zoltan Martonka
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=74304f5f2 ]
KUDU-3812 Add option to allow huge cells.
The current static 128 regular and 1024 merge row count is incompatible
with huge cells.
If we would like to allow huge cells (up to 1MB), then we have to be able
to change these values dynamically. However, we must do it in a way that
does not affect the performance of read and write operations against
tables with regular/smaller cells.
This is not trivial without having any info on cell sizes previously.
Decreasing the batch size unconditionally would hit the performance of
small data tablets. If we try to do it dynamically, then there is the
following problem:
+ Imagine we have a table schema:
- pk : int
- s1 : string <-contains 50kb strings
- s2 : string <-contains mostly 50kb, but some 1 MB strings.
+ We send a scan request, which does the following:
- we determine the 128 pks.
- we read 128 values of s1 from the columnar data. This is OK by current
logic, no need to downsize.
- we read some (let's say 20) small values of s2, until we suddenly
encounter multiple 1 MB cells.
At this point we already have around 7 MB data loaded into the server's
memory. No matter how we restructure the code at this point, if we downsize
the sent back batch to x = 21-22 rows, we have to either:
1. Throw away 128 - x values of s1 from the server's memory, so we will
fetch them from disk during the next continued scan request again.
2. Keep the 128 - x values of s1 in the server's memory. Attached to the
scan session.
Neither of them is an acceptable solution. 2. is bad, because we can have
many scan sessions, so keeping things in memory between batches can
increase risk of OOM condition.
One solution is to allow for an automatic batch sizing option of tables,
and also allow setting a higher cell size limit on tables, where the option
is turned on.
Each tablet gets an optional batch size upper limit, that can be
automatically updated at write time.
If a tablet does not have an override, the only performance impact is a
quick check for the attribute (only a single int atomic load).
If it has an upper limit for batching, it will be applied both on regular
and merge buffer sizes.
The original buffer sizes differ, because merge buffer was tuned more with
small rows in mind, while the regular buffer size is basically:
--scanner_max_batch_size_bytes / --max_cell_size_bytes = 128
However, when rows are starting to get big, using the same limit makes
sense.
This reduces memory usage in TestRandomUpsertOrderedScan from 18 GB to
9.8 GB.
Decreasing memory used over the limit from ~10 GB to 1.8 GB.
Remaining TODOs:
+ Handle update row operations:
Updates might contain values only for some of the columns. Handling it on
a best effort basis is straightforward. But with tests, it is another
hundreds of lines of code. So there will be a follow up patch for it.
+ Allow for auto batch sizing to be turned off and the batch size to be
decreased:
This is a risky task due to the number of ways log replay can go wrong.
It is indeed worth a separate commit.
Change-Id: I8a736933cbe58b7a174fac629b0e8b3d3bd72c18
Reviewed-on: https://gerrit.cloudera.org/24901
Reviewed-by: Marton Greber <[email protected]>
Reviewed-by: Alexey Serbin <[email protected]>
Tested-by: Zoltan Martonka <[email protected]>
> Fix ordered scan OOM crash in case of 1MB cells.
> ------------------------------------------------
>
> Key: KUDU-3812
> URL: https://issues.apache.org/jira/browse/KUDU-3812
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Martonka Zoltan
> Priority: Major
>
> If we just start a normal Kudu cluster and do the following:
> + Increase cell limit to 1MB.
> + Increase rpc_max_message_size, so reads does not fail with it.
> (also tablet_transaction_memory_limit_mb, so flag validator does not
> fail).
> + Create a table and start to put serious read and write load on it.
> It will quickly crash with OOM issues. It has 3 identified problems:
> + Writes and heartbeats take way too much memory if the write buffer size
> is increased. This is not really a problem, because we can keep the
> default in 7 GB. And just send few rows at once.
> + Compaction takes up way too much memory due to fixed row count.
> + Ordered scans can take up a whole GB, that is not counted towards memory
> tracking.
> If memory is already kept near hard limit (80% of system memory by default)
> by compaction or flushing, then this can easily cause a 8-10GB load with
> the default 10 rpc. Which means we can theoretically crash kudu on proper
> server hardware too (e.g 64 GB memory), not just the bare minimum
> recommended hardware.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)