[
https://issues.apache.org/jira/browse/HADOOP-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500783
]
Raghu Angadi commented on HADOOP-1134:
--------------------------------------
>> only notable code replication I see is retry logic in
>> FSInputChecker.readBuffer() where SeekToNewSource() and
>> reportChecksumFailure are executed
> Which is some of the most delicate code, that has taken several revisions to
> get to its current level of correctness. In other words, logic that shouldn't
> be replicated if at all possible.
Of course, code reuse is good. But in this case we need to write equally
important tricky and logic (in more than one place) to support sharing of
another piece of tricky code. But this is probably considered better since
there is no code replication.
Also most Jira's filed for ChecksumFileSystem are to do with the complication
of maintaining two independent streams that are related to each other by
offsets. Also each stream need to be retried correctly with different blocks.
It is complecated but Block Level CRCs does not have that problem since it only
needs to deal with one stream and one type of retry. But new Jira will be filed
any way.
> Block level CRCs in HDFS
> ------------------------
>
> Key: HADOOP-1134
> URL: https://issues.apache.org/jira/browse/HADOOP-1134
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Attachments: bc-no-upgrade-05302007.patch,
> DfsBlockCrcDesign-05305007.htm
>
>
> Currently CRCs are handled at FileSystem level and are transparent to core
> HDFS. See recent improvement HADOOP-928 ( that can add checksums to a given
> filesystem ) regd more about it. Though this served us well there a few
> disadvantages :
> 1) This doubles namespace in HDFS ( or other filesystem implementations ). In
> many cases, it nearly doubles the number of blocks. Taking namenode out of
> CRCs would nearly double namespace performance both in terms of CPU and
> memory.
> 2) Since CRCs are transparent to HDFS, it can not actively detect corrupted
> blocks. With block level CRCs, Datanode can periodically verify the checksums
> and report corruptions to namnode such that name replicas can be created.
> We propose to have CRCs maintained for all HDFS data in much the same way as
> in GFS. I will update the jira with detailed requirements and design. This
> will include same guarantees provided by current implementation and will
> include a upgrade of current data.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
