[
https://issues.apache.org/jira/browse/HDDS-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17402885#comment-17402885
]
István Fajth commented on HDDS-5447:
------------------------------------
{quote}
This approach provides an easy way to start WebHDFS together with Ozone without
installing HDFS, but doesn't require duplicated effort to maintain the code.
{quote}
I tried to enlist a few cons and pros against this statement, but later on I
realized that I really like the end goal of your suggestion, while what I am
really arguing against is the way you suggest to get there.
My main aim is to have something that works, have it soon, and have the ability
to figure out later on how and if we need to optimize the code for Ozone, while
your main aim (if I understand well) is to have less burden on the Ozone code,
and the dev community and avoid maintaining something that can be generalized
in Hadoop (where it still have to be maintained, but at least just once).
My problem is that, I find it extremely difficult to decide now if the
FileSystem based implementation is enough, if it is working for Ozone in a
performant way inside HTTPFS, and on the other hand at what point there might
be a need for implementors of the FileSystem API to hook specific things into
the generic standalone REST server implementation. That said, having the code
copied, seems way more easier, for figuring it out, than to do a generic
implementation first, that we can try out with Ozone later on (possibly after a
Hadoop release). There is too much uncertainty in my head about the final
implementation, and I do not want to take on the possibility of having to do a
new Hadoop build before a change in Ozone, and potentially a new Hadoop release
before a change can be released in Ozone.
So what if we go forward with having it in Ozone now, figure out how to
optimize and generalize it, and if it still fits into the Hadoop common project
we move it there when it is stabilized at least somewhat, instead of
generalizing in Hadoop first?
There is one more thing in the back of my head, when I suggest this... I have a
gut feeling that if we go through a few things in both projects separately, we
might figure out a better generalization of the API that is required by HTTPFS,
and for which we can add a default implementation via the FileSystem API, and
even in both project an RPC based implementation for HDFS/Ozone implementation.
What do you think [~elek]?
> HttpFS support in Ozone
> -----------------------
>
> Key: HDDS-5447
> URL: https://issues.apache.org/jira/browse/HDDS-5447
> Project: Apache Ozone
> Issue Type: New Feature
> Components: Ozone Client, Ozone Manager
> Reporter: Aravindan Vijayan
> Assignee: István Fajth
> Priority: Major
> Attachments: HTTPFS interface for Ozone.pdf
>
>
> There are several tools out there mainly written in Python, that uses the
> webhdfs interface to connect to HDFS. Even there are quite a few other
> filesystem implementations that provide access via the same rest interface
> that HDFS provides.
> HUE also implements the HDFS file browser by accessing HDFS via the REST API
> either on the NameNodes or on HTTPFS Server instances added to the HDFS
> service.
> This gave the inspiration to check and experiment what is required to support
> a similar REST endpoint over Ozone.
> The advantage is that we can ease the migration of tools developed in-house
> that are using this interface of HDFS, while we can add the possibility to
> browse Ozone from HUE.
> There is literally no disadvantage of having such an interface, as we can
> implement it as a separate module which does not have any interference with
> the rest of the code, as the REST endpoint as with HTTPFS will use a regular
> Java based Ozone client to serve any requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
