[
https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260017#comment-16260017
]
Anu Engineer commented on HDFS-7240:
------------------------------------
h1. Ozone - First community meeting
{{Time: Thursday, November 16, 2017, at 1:00:00 am PST}}
_Participants: Anu Engineer, Mukul Kumar Singh, Nandakumar Vadivelu, Weiwei
Yang, Steve Loughran, Thomas Demoor, Shashikant Banerjee, Lokesh Jain_
We discussed quite a large number of technical issues at this meeting.
We went over how Ozone's works, the Namespace architecture of KSM and how it
interacts with SCM. We traced both a write I/O path and read I/O path.
There was some discussion over the REST protocol and making sure that Rest
protocol is good enough to support Hadoop based workloads. We look at various
REST APIs of Ozone and also discussed O3 FS working over RPC instead of REST
protocol. This is a work in progress.
Steve Loughran suggested that we add Storm to the applications that are tested
against Ozone. Currently, we use Hive, Spark, YARN, as the applications to test
against Ozone. We will add Storm to this testing mix.
We discussed performance and scale of testing; ozone has been tested with
millions of keys. We have also tested with cluster sizes up to 300 nodes.
Steve suggested that we upgrade the Ratis version and lock that down before the
merge.
Thomas Demoor pointed out the difference between the commit ordering of S3 and
Ozone. Ozone uses the actual commit time to decide the key ordering, S3 uses
the key creation time to decide the ordering of the keys. He also mentioned
that this should not matter in the real world as he is not aware hard-coded
dependency on commit ordering.
> Object store in HDFS
> --------------------
>
> Key: HDFS-7240
> URL: https://issues.apache.org/jira/browse/HDFS-7240
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
> Attachments: HDFS Scalability and Ozone.pdf, HDFS-7240.001.patch,
> HDFS-7240.002.patch, HDFS-7240.003.patch, HDFS-7240.003.patch,
> HDFS-7240.004.patch, HDFS-7240.005.patch, HDFS-7240.006.patch,
> MeetingMinutes.pdf, Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf,
> ozone_user_v0.pdf
>
>
> This jira proposes to add object store capabilities into HDFS.
> As part of the federation work (HDFS-1052) we separated block storage as a
> generic storage layer. Using the Block Pool abstraction, new kinds of
> namespaces can be built on top of the storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode
> storage, but independent of namespace metadata.
> I will soon update with a detailed design document.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
