[
https://issues.apache.org/jira/browse/HDDS-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397720#comment-17397720
]
István Fajth commented on HDDS-5447:
------------------------------------
Hi [~elek],
thank you for taking our attention to the mentioned JIRAs.
I have attached the design doc, hopefully it addresses your concerns, but let
me add some short answers for you here as well.
This interface - unlike the one I understood we removed as part of HDDS-738 -
intends to implement a common API that is used by tools to access data on HDFS.
With that, in nature and probable maintenance cost, it is similar to what we
have with S3 gateway, but provides an other commonly known way to access an
Ozone cluster's data.
WebHDFS REST API is a well established API that uses the Hadoop FileSystem API
to access anything that has a FileSystem API. A Hadoop HTTPFS server can be
configured even today to use ofs:// file system scheme to connect to Ozone, it
is just hard to manage and pull in a good amount of HDFS dependencies, and use
HDFS lifecycle scripts to configure and run it, if you are managing an Ozone
cluster, hence the idea to pull it into Ozone, and provide the necessary
infrastructure inside the project to run it easily.
The WebHDFS interface is used by a couple of tools, written in other languages
like python to access HDFS data on existing clusters. The main motivation is to
provide this subset of the API as an official component in Ozone, so that those
HDFS operators who's users are relying on the WebHDFS API for some reason to
access data have an easily applicable alternative for those users/tools.
Management operations we can decide later on as we start to get support for
things like EC policies, but I think if we want to reduce the maintenance cost,
we should just leave endpoints like those unimplemented in our project via the
WebHDFS interface, and for those things rely on operators, and the otherwise
available admin tools.
> HttpFS support in Ozone
> -----------------------
>
> Key: HDDS-5447
> URL: https://issues.apache.org/jira/browse/HDDS-5447
> Project: Apache Ozone
> Issue Type: New Feature
> Components: Ozone Client, Ozone Manager
> Reporter: Aravindan Vijayan
> Assignee: István Fajth
> Priority: Major
> Attachments: HTTPFS interface for Ozone.pdf
>
>
> There are several tools out there mainly written in Python, that uses the
> webhdfs interface to connect to HDFS. Even there are quite a few other
> filesystem implementations that provide access via the same rest interface
> that HDFS provides.
> HUE also implements the HDFS file browser by accessing HDFS via the REST API
> either on the NameNodes or on HTTPFS Server instances added to the HDFS
> service.
> This gave the inspiration to check and experiment what is required to support
> a similar REST endpoint over Ozone.
> The advantage is that we can ease the migration of tools developed in-house
> that are using this interface of HDFS, while we can add the possibility to
> browse Ozone from HUE.
> There is literally no disadvantage of having such an interface, as we can
> implement it as a separate module which does not have any interference with
> the rest of the code, as the REST endpoint as with HTTPFS will use a regular
> Java based Ozone client to serve any requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
