[
https://issues.apache.org/jira/browse/HDDS-2010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Cheng updated HDDS-2010:
---------------------------
Fix Version/s: 0.5.0
> PipelineID management for multi-raft, in SCM or in datanode?
> ------------------------------------------------------------
>
> Key: HDDS-2010
> URL: https://issues.apache.org/jira/browse/HDDS-2010
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
> Reporter: Li Cheng
> Assignee: Li Cheng
> Priority: Major
> Fix For: 0.5.0
>
>
> With the intention to support
> [multi-raft|https://issues.apache.org/jira/browse/HDDS-1564], I wanna bring
> up a question on how the pipeline unique ids be managed. Since every datanode
> can be member in multiple raft pipelines, the pipeline ids need to be
> persisted with the datanode for recovery purpose (we can talk about recovery
> later). Generally there are two options:
> # Store in datanode (like datanodeDetails) and every time pipelines mapping
> change on single datanode, pipeline ids will be serialized to local file.
> This way will lead to many more local serialization of things like
> datanodeDetails, but the updates are only for local datanode change.
> Improvement can be made like linking a serializable object to datanodeDetails
> and datanode keeps updating the new pipeline ids to the serializable object
> instead the details file. On the other hand, since the pipeline ids are
> stored only in datanode locally, there will be no global view in SCM. (or we
> can store a lazy copy?)
> * 2. Stored in SCM. SCM can maintain a large mapping between datanode ids
> and pipeline ids. But this way will lead to an exponentially increasing
> frequency in SCM updates since the pipeline mapping changes are way more
> complex and happen all the time. Obviously this gives SCM too much pressure,
> but it can also give SCM a global view on the management over datanodes and
> multi raft pipelines.
>
> Thoughts? [~xyao] [~Sammi]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
