[
https://issues.apache.org/jira/browse/KUDU-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18087035#comment-18087035
]
ASF subversion and git services commented on KUDU-3429:
-------------------------------------------------------
Commit 3638b3814e36cbcb90ebfbf02dbbfaf7a0d70843 in kudu's branch
refs/heads/master from Ashwani
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=3638b3814 ]
KUDU-3429 Make rows per block flag based (1/n)
Convert hard-coded kRowsPerBlock to a flag based variable that can be
used and modified by tests at runtime to control memory usage during
rowset compaction.
MajorDeltaCompaction and MinorDeltaCompaction also have their own
kRowsPerBlock. The patch doesn't modify those to flags.
This changelist doesn't contain any functional modifications.
Change-Id: I65f1e0d91e718b097bcc6e3979f246a13a618edf
Reviewed-on: http://gerrit.cloudera.org:8080/24328
Reviewed-by: Marton Greber <[email protected]>
Tested-by: Marton Greber <[email protected]>
> Refactor CompactRowSetsOp to run on a pre-determined memory budget
> -------------------------------------------------------------------
>
> Key: KUDU-3429
> URL: https://issues.apache.org/jira/browse/KUDU-3429
> Project: Kudu
> Issue Type: Improvement
> Reporter: Alexey Serbin
> Assignee: Ashwani Raina
> Priority: Major
>
> [KUDU-3406|https://issues.apache.org/jira/browse/KUDU-3406] added memory
> budgeting for running CompactRowSetsOp maintenance operations. On its
> nature, that provides an interim approach adding memory budgeting on top of
> the current CompactRowSetsOp implementation as-is.
> Ideally, the implementation of CompactRowSetsOp should be refactored to merge
> the deltas in participating rowsets sequentially, chunk by chunk, persisting
> the results and allocating memory just for small bunch of processed deltas,
> not loading all the deltas at once.
> This JIRA item is to track the work in the context outlined above.
> Key points to address in this scope:
> * even if it's a merge-like operation by its nature, the current
> implementation of CompactRowSetsOp allocates all the memory necessary to load
> the UNDO deltas at once, and it keeps all the preliminary results in the
> memory as well before persisting the result data to disk
> * the current implementation of CompactRowSetsOp loads all the UNDO deltas
> from the rowsets selected for compaction regardless whether they are ancient
> or not; it discards of the data sourced from the ancient deltas in the very
> end before persisting the result data
> Also, while keeping memory usage on a predetermined budget, the new
> implementation for CompactRowSetsOp should strive to avoid IO multiplication
> as much as possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
