[
https://issues.apache.org/jira/browse/HDFS-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-10759:
------------------------------
Attachment: HDFS-10759.0002.patch
Attached is HDFS-10759.0002.patch which will make this change backwards
compatible to the existing {{boolean isStriped}} in the {{fsimage}}. This is
done by making the {{BlockType}} enum work with {{CONTIGUOUS=0, STRIPED=1}} as
this mirrors the boolean behaviour.
As both the boolean and the enum are optional, they are usually left out of the
wire protocol. If they are explicitly set then they will have the same semantic
values on the wire now.
Given the following messages:
{code}
/**
* Types of recognized blocks.
*/
enum BlockTypeProto {
CONTIGUOUS = 0;
STRIPED = 1;
}
message INodeFile {
optional uint32 replication = 1;
optional uint64 modificationTime = 2;
optional uint64 accessTime = 3;
optional uint64 preferredBlockSize = 4;
optional fixed64 permission = 5;
repeated BlockProto blocks = 6;
optional FileUnderConstructionFeature fileUC = 7;
optional AclFeatureProto acl = 8;
optional XAttrFeatureProto xAttrs = 9;
optional uint32 storagePolicyID = 10;
optional BlockTypeProto blockType = 11;
}
/* ehiggs - old style using bool blockType */
message INodeFileOld {
optional uint32 replication = 1;
optional uint64 modificationTime = 2;
optional uint64 accessTime = 3;
optional uint64 preferredBlockSize = 4;
optional fixed64 permission = 5;
repeated BlockProto blocks = 6;
optional FileUnderConstructionFeature fileUC = 7;
optional AclFeatureProto acl = 8;
optional XAttrFeatureProto xAttrs = 9;
optional uint32 storagePolicyID = 10;
optional bool isStriped = 11;
}
{code}
We can then see that these serialise as the same values:
{code}
In [1]: import fsimage_pb2
In [2]: f_enum = fsimage_pb2.INodeSection.INodeFile()
In [3]: f_bool = fsimage_pb2.INodeSection.INodeFileOld()
In [4]: f_enum.SerializeToString() # Wire format of entirely optional message.
Empty!
Out[4]: ''
In [5]: f_bool.SerializeToString() # With a bool, everything is still empty. No
surprises.
Out[5]: ''
In [6]: f_enum.blockType = 0
In [7]: f_bool.isStriped = False
In [8]: f_enum.SerializeToString() # wire format of explicit
BlockType.CONTIGUOUS
Out[8]: 'X\x00'
In [9]: f_bool.SerializeToString() # wire format of explicit false
Out[9]: 'X\x00'
In [10]: f_enum.blockType = 1 # Set explicity to STRIPED
In [11]: f_bool.isStriped = True # turn isStriped to True.
In [12]: f_enum.SerializeToString()
Out[12]: 'X\x01'
In [13]: f_bool.SerializeToString()
Out[13]: 'X\x01'
{code}
> Change fsimage bool isStriped from boolean to an enum
> -----------------------------------------------------
>
> Key: HDFS-10759
> URL: https://issues.apache.org/jira/browse/HDFS-10759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2
> Reporter: Ewan Higgs
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10759.0001.patch, HDFS-10759.0002.patch
>
>
> The new erasure coding project has updated the protocol for fsimage such that
> the {{INodeFile}} has a boolean '{{isStriped}}'. I think this is better as an
> enum or integer since a boolean precludes any future block types.
> For example:
> {code}
> enum BlockType {
> CONTIGUOUS = 0,
> STRIPED = 1,
> }
> {code}
> We can also make this more robust to future changes where there are different
> block types supported in a staged rollout. Here, we would use
> {{UNKNOWN_BLOCK_TYPE}} as the first value since this is the default value.
> See
> [here|http://androiddevblog.com/protocol-buffers-pitfall-adding-enum-values/]
> for more discussion.
> {code}
> enum BlockType {
> UNKNOWN_BLOCK_TYPE = 0,
> CONTIGUOUS = 1,
> STRIPED = 2,
> }
> {code}
> But I'm not convinced this is necessary since there are other enums that
> don't use this approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
