[
https://issues.apache.org/jira/browse/HDFS-9938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192586#comment-15192586
]
Chris Douglas commented on HDFS-9938:
-------------------------------------
As
[discussed|https://issues.apache.org/jira/browse/HADOOP-12666?focusedCommentId=15186380&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15186380]
in the related JIRA, extending WebHDFS this way is undesirable. This
connection will be
[severed|https://issues.apache.org/jira/browse/HADOOP-12666?focusedCommentId=15189533&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15189533]
as the implementation evolves, implementing approach (3).
Locating part of the implementation in the same package (approach (1) in the
description) is not hygienic, but it's not without precedent (e.g., Mumak,
yarn-server/yarn-server-common, the fault injection tooling). For reviewers
unfamiliar with this history, it's also clear that future issues are the fault
of the ADL implementation overriding package-private methods, and this
extension was not part of the WebHDFS design. If ADL impedes the development of
WebHDFS, then it would be permissible to delete it from the repository, as has
been the fate of similar modules.
Looking through the details of the patch (approach (2)): the visibility of the
{{o.a.h.hdfs.web.\*Param}} classes is inconsistent, so those changes are mostly
neutral. However, relaxing visibility in {{WebHdfsFileSystem}} is similar to
approaches rejected elsewhere in HDFS. There must be an API that defines a
contract for extensions, and making methods public is insufficient. HDFS-8154
is an example of how we _should_ define protocol compatibility for WebHDFS.
Short of that, refactoring WebHDFS to support overriding some internal details
might allow multiple implementations to reuse its machinery without ill
effects, such as colliding configuration variables (as [~mackrorysd] and others
pointed out in HADOOP-12666).
A short-term implementation of approach (3) would copy the machinery of WebHDFS
into the ADL package. This seems strictly worse than the other approaches.
I favor approach (1), unless there's a cleaner design for extension. Given the
review feedback on HADOOP-12666, extending WebHDFS at all should be a stopgap
measure.
Minor: the patch accidentally contains some pom modifications from
HADOOP-12666. Also, would {{protected}} be sufficient for most of the
modifications to {{WebHdfsFileSystem}}?
> Sub classing of WebHDFS for the new file system implementation
> --------------------------------------------------------------
>
> Key: HDFS-9938
> URL: https://issues.apache.org/jira/browse/HDFS-9938
> Project: Hadoop HDFS
> Issue Type: Wish
> Reporter: Vishwajeet Dusane
> Attachments: HDFS-9938-001.patch
>
>
> This JIRA is to request opinion from the community, Can new file system use
> an extension of WebHDFS file system.
> Considering all the known limitation WebHDFS has over implementing a new
> client by extending FileSystem class.
> Option we have is
> 1. Use the namespace org.apache.hadoop.hdfs.web in new file system
> implementation to access protected functionality from WebHdfsFileSystem.
> 2. Change the WebHdfs to support extensions
> 3. Suggestion on different approach like new client by sub classing
> FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
