[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576683#comment-14576683
]
Kai Zheng commented on HDFS-8494:
---------------------------------
Comments for the latest patch:
1. To avoid the overhead of {{getErasureCodingZone}} for each
{{StripedDataStreamer}} in the following change, please pass
{{ErasureCodingZone}} in the constructor of StripedDataStreamer, sharing the
same instance of ErasureCodingZone for all the needed streamers.
{code}
void putLoactedBlocks(LocatedBlock lb) throws IOException {
+ ErasureCodingZone ecZone = dfsClient.getErasureCodingZone(src);
if (LOG.isDebugEnabled()) {
LOG.debug("Obtained block group " + lb);
}
LocatedBlock[] blocks = StripedBlockUtil.parseStripedBlockGroup(
(LocatedStripedBlock)lb,
- BLOCK_STRIPED_CELL_SIZE, NUM_DATA_BLOCKS, NUM_PARITY_BLOCKS);
+ ecZone.getCellSize(), NUM_DATA_BLOCKS, NUM_PARITY_BLOCKS);
{code}
2. In balancer/Dispatcher, need to investigate how to remove the hard-coded
value. I would open separate JIRA for this as follow-on.
{code}
- HdfsConstants.BLOCK_STRIPED_CELL_SIZE, dataBlockNum, idxInGroup);
+ HdfsConstants.DEFAULT_BLOCK_STRIPED_CELL_SIZE, dataBlockNum,
idxInGroup);
{code}
3. Please avoid the 2nd call to {{getCellSize}}.
{code}
+ int cellSize = getCellSize(sblock);
+ if (cellSize > 0) {
+ dataBlockNum = (short) Math.min(dataBlockNum,
+ (sblock.getNumBytes() - 1) / getCellSize(sblock) + 1);
+ }
{code}
> Remove hard-coded chunk size in favor of ECZone
> -----------------------------------------------
>
> Key: HDFS-8494
> URL: https://issues.apache.org/jira/browse/HDFS-8494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: HDFS-7285
> Reporter: Kai Sasaki
> Assignee: Kai Sasaki
> Fix For: HDFS-7285
>
> Attachments: HDFS-8494-HDFS-7285-01.patch,
> HDFS-8494-HDFS-7285-02.patch
>
>
> It is necessary to remove hard-coded values inside NameNode configured in
> {{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
> favor of HDFS-8375.
> Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
> {{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
> when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
