[
https://issues.apache.org/jira/browse/KUDU-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yingchun Lai updated KUDU-3371:
-------------------------------
Description:
h1. motivation
The current LBM container use separate .data and .metadata files. The .data
file store the real user data, we can use hole punching to reduce disk space.
While the metadata use write protobuf serialized string to a file, in append
only mode. Each protobuf object is a struct of BlockRecordPB:
{code:java}
message BlockRecordPB {
required BlockIdPB block_id = 1; // int64
required BlockRecordType op_type = 2; // CREATE or DELETE
required uint64 timestamp_us = 3;
optional int64 offset = 4; // Required for CREATE.
optional int64 length = 5; // Required for CREATE.
} {code}
That means each object is either type of CREATE or DELETE. To mark a 'block' as
deleted, there will be 2 objects in the metadata, one is CREATE type and the
other is DELETE type.
There are some weak points of current LBM metadata storage mechanism:
h2. 1. Disk space amplification
h2. 2. Long time bootstrap
> Use RocksDB to store LBM metadata
> ---------------------------------
>
> Key: KUDU-3371
> URL: https://issues.apache.org/jira/browse/KUDU-3371
> Project: Kudu
> Issue Type: Improvement
> Components: fs
> Reporter: Yingchun Lai
> Priority: Major
>
> h1. motivation
> The current LBM container use separate .data and .metadata files. The .data
> file store the real user data, we can use hole punching to reduce disk space.
> While the metadata use write protobuf serialized string to a file, in append
> only mode. Each protobuf object is a struct of BlockRecordPB:
>
> {code:java}
> message BlockRecordPB {
> required BlockIdPB block_id = 1; // int64
> required BlockRecordType op_type = 2; // CREATE or DELETE
> required uint64 timestamp_us = 3;
> optional int64 offset = 4; // Required for CREATE.
> optional int64 length = 5; // Required for CREATE.
> } {code}
> That means each object is either type of CREATE or DELETE. To mark a 'block'
> as deleted, there will be 2 objects in the metadata, one is CREATE type and
> the other is DELETE type.
> There are some weak points of current LBM metadata storage mechanism:
> h2. 1. Disk space amplification
>
> h2. 2. Long time bootstrap
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
