[
https://issues.apache.org/jira/browse/HDFS-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883481#action_12883481
]
Rodrigo Schmidt commented on HDFS-1111:
---------------------------------------
The RaidNode is not currently using this API, although its use was one of the
motivations I had for adding getCorruptFiles() to ClientProtocol. Originally,
raid was part of HDFS and I could certainly see how Raid (and possibly other
parts of HDFS) could benefit from this as an RPC to the namenode. I thought the
others saw it too because when I got to HDFS-729, having getCorruptFiles() on
ClientProtocol was not under discussion anymore.
The JIRA that is responsible for making the RaidNode call getCorruptFiles is
HDFS-1171. Most probably we will have to extend DistributedFileSystem to export
getCorruptFiles(). That's why I said we don't have to be external to HDFS, but
we can be external to the namenode.
On the other hand, if we do take getCorruptFiles() out of ClientProtocol, we
will make HDFS-1171 overly complicated or expensive.
I really think the correct design choice is to export basic APIs like
getCorruptFiles() as RPCs and build services like fsck and raid completely
outside the namenode. After looking at the fsck code from the inside out and
having experienced how it can sometimes compromise the whole filesystem because
the namenode is using most of its resources to calculate outputs for fsck
requests, I'm convinced it should be outside the namenode. For the sake of
horizontal scalability of the namenode, we should be working in redesigning
things like the current fsck implementation, instead of reinforcing it.
That's what I meant when I said we should be taking things out of the namenode.
In my opinion, even if my case about having other parts of HDFS call
getCorruptFiles() is not convincing, taking it out of ClientProtocol only
reinforces the design choice of running fsck inside the namenode, which I think
is bad. As we have more and more discussions about a distributed namenode,
things like fsck should be the first ones running externally to it (to the
namenode, not to HDFS). I see this as a low-hanging fruit towards a more
scalable and distributed namenode.
> getCorruptFiles() should give some hint that the list is not complete
> ---------------------------------------------------------------------
>
> Key: HDFS-1111
> URL: https://issues.apache.org/jira/browse/HDFS-1111
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Rodrigo Schmidt
> Assignee: Rodrigo Schmidt
> Attachments: HADFS-1111.0.patch
>
>
> If the list of corruptfiles returned by the namenode doesn't say anything if
> the number of corrupted files is larger than the call output limit (which
> means the list is not complete). There should be a way to hint incompleteness
> to clients.
> A simple hack would be to add an extra entry to the array returned with the
> value null. Clients could interpret this as a sign that there are other
> corrupt files in the system.
> We should also do some rephrasing of the fsck output to make it more
> confident when the list is not complete and less confident when the list is
> known to be incomplete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
