[
https://issues.apache.org/jira/browse/HDDS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Rose updated HDDS-3869:
-----------------------------
Description:
Currently datanodes place all of their data under the default column family in
RocksDB. This differs from OM and SCM which organize their data into different
column families based on its type. This feature will first move the datanode
code off of the database utilities in the hadoop.hdds.utils package (which has
no column family support), and move them to the newer utilities used by OM and
SCM in the hadoop.hdds.utils.db package (which has column family support). The
datanode will divide its data into three column families:
# block_data: String keys (block id with optional prefix) map to BlockData
objects
# metadata: String keys (name of metadata field) map to Long objects.
# deleted_blocks: String keys (block id with optional prefix) map to the
ChunkInfo lists (lists of chunks corresponding to the block that was deleted).
A new field, called 'schemaVersion' will be added to container files to
indicate whether the container was created using the original schema version 1,
where everything was in the default column family, or this new schema version
2. Code should be able to process older schema versions for backwards
compatibility.
was:Currently datanodes place all of their data under the default column
family in RocksDB. This differs from OM and SCM which organize their data into
different column families based on its type. By having the datanode place its
metadata in one column family, and block data in another, it will be easier to
view for debugging and organizational purposes.
> Use different column families for datanode block and metadata
> -------------------------------------------------------------
>
> Key: HDDS-3869
> URL: https://issues.apache.org/jira/browse/HDDS-3869
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
> Reporter: Ethan Rose
> Assignee: Ethan Rose
> Priority: Critical
> Labels: upgrade
>
> Currently datanodes place all of their data under the default column family
> in RocksDB. This differs from OM and SCM which organize their data into
> different column families based on its type. This feature will first move the
> datanode code off of the database utilities in the hadoop.hdds.utils package
> (which has no column family support), and move them to the newer utilities
> used by OM and SCM in the hadoop.hdds.utils.db package (which has column
> family support). The datanode will divide its data into three column families:
> # block_data: String keys (block id with optional prefix) map to BlockData
> objects
> # metadata: String keys (name of metadata field) map to Long objects.
> # deleted_blocks: String keys (block id with optional prefix) map to the
> ChunkInfo lists (lists of chunks corresponding to the block that was deleted).
> A new field, called 'schemaVersion' will be added to container files to
> indicate whether the container was created using the original schema version
> 1, where everything was in the default column family, or this new schema
> version 2. Code should be able to process older schema versions for backwards
> compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
