[
https://issues.apache.org/jira/browse/HDFS-10867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016962#comment-16016962
]
Chris Douglas commented on HDFS-10867:
--------------------------------------
We had an offline discussion about this, and noticed the following
refinement/alternative to fixing the bits allocated to provided storage. In the
doc, of the 64 bits allocated per BlockID, the two high-level bits are
allocated to a "block type" which defines the semantics for the remaining bits.
For provided blocks, 10 bits are allocated to a partition of BlockIDs within a
mount. Reproducing the table:
|| Block Type || Payload || ||
| Contiguous (0b00) | First 2^30 reserved, (30 bit) | |
| Striped (0b10) | | |
| Provided (0b01) | Provided Storage ID (10 bit) | Block ID |
Instead of separating the block type, we considered a scheme that defines the
existing, {{CONTIGUOUS}} and {{STRIPED}} schemes as partitions of the block
space. {{PROVIDED}} mounts would consume partitions as mounts are created, in
the following scheme:
|| Block partition || BlockID ||
| reverse(0bPARTITION) | ... |
So {{CONTIGUOUS}} (0x0) and {{STRIPED}} (0x1) are merely the first allocated
partitions. Subsequent partitions (0x01, 0x11, 0x001, ...) may consume the 2+10
bits of the fixed scheme if 4096 are created, but the number of partitions and
BlockIDs could float as required. Partitions could be reused, if the last
allocated BlockID is persisted and disjoint across mounts.
> Block Bit Field Allocation of Provided Storage
> ----------------------------------------------
>
> Key: HDFS-10867
> URL: https://issues.apache.org/jira/browse/HDFS-10867
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
> Reporter: Ewan Higgs
> Attachments: Block Bit Field Allocation of Provided Storage.pdf
>
>
> We wish to design and implement the following related features for provided
> storage:
> # Dynamic mounting of provided storage within a Namenode (mount, unmount)
> # Mount multiple provided storage systems on a single Namenode.
> # Support updates to the provided storage system without having to regenerate
> an fsimg.
> A mount in the namespace addresses a corresponding set of block data. When
> unmounted, any block data associated with the mount becomes invalid and
> (eventually) unaddressable in HDFS. As with erasure-coded blocks, efficient
> unmounting requires that all blocks with that attribute be identifiable by
> the block management layer
> In this subtask, we focus on changes and conventions to the block management
> layer. Namespace operations are covered in a separate subtask.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
