Zoltan Martonka has uploaded this change for review. (
http://gerrit.cloudera.org:8080/24901
Change subject: WIP KUDU-2874 Add option to allow huge cells.
......................................................................
WIP KUDU-2874 Add option to allow huge cells.
The current static 128 normal and 1024 merge row count is incompatible
with huge cells.
If we would like to allow huge cells (up to 1MB), then we have to be
able to change this values dynamically. However we must do it in a
way that does not affect the performance of already existing tables.
This is not trivial without having any info on cell sizes previously.
Decreasing the batch size unconditionally would hit the performance of
small data tablets. If we try to do it dynamically, then there is the
following problem:
+ Imagine we have a tablet:
- pk : int
- s1 : string <-contains 50kb strings
- s2 : string <-contains mostly 50kb, but some 1 MB strings.
+ We send a scan request, which does the following:
- we determine the 128 pks.
- we read 128 values of s1 from the columnar data. This is ok by current
logic, no need to downsize.
- we read some (lets say 20) small values of s2, until we suddenly encounter
multiple 1 MB cells.
At this point we already have around 7 MB data loaded. No matter how we
restructure the code at this point if we downsize the sent back batch to
x = 21-22 rows, we have to either:
1. Throw away 128 - x values of s1, so we will reread them in next batch
2. Keep the 128- x values of s1 in the memory attached to the scan session.
Neither of them is an acceptable solution. 2. is bad, because we can have many
scan session, so keeping things in memory between batches can increase risk of
OOM.
One solution is to allow for a "huge" option of tablets. If a tablet is
not marked as "huge", the only performance impact is a quick check for
the attribute.
If it is marked, we allow cells up to 1MB, however we decrease the
batch size used in case of this tablet at write time if any big cells is
committed to the tablet.
This reduces memory usage in TestRandomUpsertOrderedScan from 26GB to:
Peak tmalloc memory during test : 13825 MiB
Peak root mem tracker memory during test : 13126 MiB
Decreasing memory used over the limit from 18 to 5 GB.
Change-Id: I8a736933cbe58b7a174fac629b0e8b3d3bd72c18
---
M src/kudu/common/common.proto
M src/kudu/common/generic_iterators.cc
M src/kudu/common/generic_iterators.h
M src/kudu/common/row_operations.cc
M src/kudu/common/row_operations.h
M src/kudu/common/wire_protocol.cc
M src/kudu/common/wire_protocol.h
M src/kudu/tablet/local_tablet_writer.h
M src/kudu/tablet/metadata.proto
M src/kudu/tablet/ops/write_op.cc
M src/kudu/tablet/rowset.cc
M src/kudu/tablet/rowset.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tserver/ordered_scan_large_cells-test.cc
M src/kudu/tserver/tablet_service.cc
19 files changed, 373 insertions(+), 32 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/01/24901/1
--
To view, visit http://gerrit.cloudera.org:8080/24901
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8a736933cbe58b7a174fac629b0e8b3d3bd72c18
Gerrit-Change-Number: 24901
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Martonka <[email protected]>
