[
https://issues.apache.org/jira/browse/HDDS-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787555#comment-16787555
]
Aravindan Vijayan edited comment on HDDS-1232 at 3/8/19 5:54 AM:
-----------------------------------------------------------------
*A summary of what the patch contains*
A basic requirement from Recon is that it should be able to tell which keys are
part of a container. The Recon Container DB is designed to hold this
information which is basically a reverse map of the OM DB information. Instead
of storing the entire Container -> Key mapping, it stores the map of
(Container, Key Prefix) -> Count of keys that match that prefix in the
container.
For example, if a container 'C1' has the following keys
* Vol1/Bucket1/Prefix1ABC
* Vol1/Bucket1/Prefix1EFG
* Vol1/Bucket1/Prefix2DEF
* Vol1/Bucket2/Prefix1HIG
This information is stored as follows in the Recon container DB.
C1-Vol1/Bucket1/Prefix1 -> 2
C1-Vol1/Bucket1/Prefix2 -> 1
C1-Vol1/Bucket2/Prefix1 -> 1
Using the above mapping and the OM metadata DB snapshot (HDDS-1233), we can get
the mapping of Container -> Key.
The interface ContainerDBServiceProvider provides APIs to store / get this
information. The ContainerDBServiceProviderImpl is the implementation of this
interface and is instantiated through injection. It holds an instance of
RocksDB which it uses to store the above information.
This patch also contains other related changes like
* wiring up hadoop-ozone-recon to the other modules in hadoop-ozone
distribution.
* Adding testing of Recon config keys to the TestConfigurationFields utility.
* Refactor a utility method to be re-usable in
org.apache.hadoop.hdds.server.ServerUtils which attempts to get the DB dir for
a given component from config, and falls back to ozone.metadata.dirs if the
config is not present.
* Add a prefix seek iterator in the org.apache.hadoop.utils.MetaStoreIterator
wrapper around the underlying RocksDB/LevelDB iterator + corresponding unit
test.
was (Author: avijayan):
*A summary of what the patch contains*
A basic requirement from Recon is that it should be able to tell which keys are
part of a container. The Recon Container DB is designed to hold this
information which is basically a reverse map of the OM DB information. Instead
of storing the entire Container -> Key mapping, it stores the map of
(Container, Key Prefix) -> Count of keys that match that prefix in the
container.
For example, if a container 'C1' has the following keys
* Vol1/Bucket1/Prefix1ABC
* Vol1/Bucket1/Prefix1EFG
* Vol1/Bucket1/Prefix2DEF
* Vol1/Bucket2/Prefix1HIG
This information is stored as follows in the Recon container DB.
C1-Vol1/Bucket1/Prefix1 -> 2
C1-Vol1/Bucket1/Prefix2 -> 1
C1-Vol1/Bucket2/Prefix1 -> 1
Using the above mapping and the OM metadata DB snapshot (HDDS-1233), we can get
the mapping of Container -> Key and Container -> BlockId.
The interface ContainerDBServiceProvider provides APIs to store / get this
information. The ContainerDBServiceProviderImpl is the implementation of this
interface and is instantiated through injection. It holds an instance of
RocksDB which it uses to store the above information.
This patch also contains other related changes like
* wiring up hadoop-ozone-recon to the other modules in hadoop-ozone
distribution.
* Adding testing of Recon config keys to the TestConfigurationFields utility.
* Refactor a utility method to be re-usable in
org.apache.hadoop.hdds.server.ServerUtils which attempts to get the DB dir for
a given component from config, and falls back to ozone.metadata.dirs if the
config is not present.
* Add a prefix seek iterator in the org.apache.hadoop.utils.MetaStoreIterator
wrapper around the underlying RocksDB/LevelDB iterator + corresponding unit
test.
> Recon Container DB service definition
> -------------------------------------
>
> Key: HDDS-1232
> URL: https://issues.apache.org/jira/browse/HDDS-1232
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Recon
> Reporter: Aravindan Vijayan
> Assignee: Aravindan Vijayan
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: HDDS-1232-000.patch, HDDS-1232-001.patch,
> HDDS-1232-002.patch, HDDS-1232-003.patch, HDDS-1232-004.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> * Define the Ozone Recon container DB service definition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
