Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hi Sanjay, With respect to Ozone my two main concerns were: 1. Wether Ozone can help scaling out the namespace service in handling higher RPC workloads. I think we came to common conclusion that using Ozone as a block management layer is a reasonable path to scaling HDFS. The discussions are

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Konstantine Thanks for your feedback and comments over the last few months. Have we addressed all your issues and concerns? sanjay > On Feb 13, 2018, at 6:28 PM, sanjay Radia wrote: > > Sorry the formatting got messed by my email client. Here it is again > > >

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Sorry the formatting got messed by my email client. Here it is again Dear Hadoop Community Members, We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Dear Hadoop Community Members, We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments. The key questions raised were following How the new block storage

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hi Sanjay, Read your doc. I clearly see the value of Ozone with your use cases, but I agree with Stack and others the question why it should be a part of Hadoop isn't clear. More details in the jira:

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

> At a minimum, it should at least be using it’s own maven module for a > lot of the bits that generates it’s own maven jars so that we can split this > functionality up at build/test time. I expected this to be the case, but looks like it isn't. There's lot of value in splitting the

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hi folks! Thank you for sharing the design docs and the tremendous amount of work that has gone into Ozone. I'm grateful that atleast someone is trying to drastically improve HDFS. *If* there is a meeting to discuss this merge, could I please also be invited? Have we ever thought about

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Konstantine, Thanks for your comments, questions and feedback. I have attached a document to the HDFS-7240 jira that explains a design for scaling HDFS and how Ozone paves the way towards the full solution.

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

> On Nov 3, 2017, at 12:08 PM, Stack wrote: > > On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko > wrote: > >> It is an interesting question whether Ozone should be a part of Hadoop. > > I don't see a direct answer to this question. Is there one?

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

t; design doc<https://issues.apache.org/jira/secure/attachment/1279954 > >> 9/ozone_user_v0.pdf> uses most common syntax so I believe it should be > >> compliance. You can find the rest API doc here<https://github.com/apache > >> /hadoop/blob/HDFS-7240

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko wrote: > Hey guys, > > It is an interesting question whether Ozone should be a part of Hadoop. > I don't see a direct answer to this question. Is there one? Pardon me if I've not seen it but I'm interested in the

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

gt;> design doc<https://issues.apache.org/jira/secure/attachment/1279954 > >> 9/ozone_user_v0.pdf> uses most common syntax so I believe it should be > >> compliance. You can find the rest API doc > here<https://github.com/apache > >> /hadoop/blob/H

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hi Konstantin, Thank you for taking out time to review ozone. I appreciate your comments and questions. > There are two main limitations in HDFS > a) The throughput of Namespace operations. Which is limited by the >number of RPCs the NameNode can handle > b) The number of objects (files +

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hey guys, It is an interesting question whether Ozone should be a part of Hadoop. There are two main reasons why I think it should not. 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a sizable community behind, it looks to me like a whole new project. It is essentially a new

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

down/OzoneRest.md> (with some example usages), and commandline >> API here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop- >> hdfs-project/hadoop-hdfs/src/site/markdown/OzoneCommandShell.md>. >> >> >> Look forward for your feedbac

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

the rest API doc here<https://github.com/apache >> /hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/ >> site/markdown/OzoneRest.md> (with some example usages), and commandline >> API here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop- >>

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

ack! > > > --Weiwei > > > > 发件人: Steve Loughran <ste...@hortonworks.com> > 发送时间: 2017年10月20日 11:49 > 收件人: Yang Weiwei > 抄送: hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; > yarn-dev@hadoop.apache.

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

@hortonworks.com> 发送时间: 2017年10月20日 11:49 收件人: Yang Weiwei 抄送: hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; common-...@hadoop.apache.org 主题: Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk Wow, big

答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

送: hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; common-...@hadoop.apache.org 主题: Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk Wow, big piece of work 1. Where is a PR/branch on github with rendered docs for us to look at? 2. Have you made

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Wow, big piece of work 1. Where is a PR/branch on github with rendered docs for us to look at? 2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details. thanks > On 19 Oct 2017, at 02:54, Yang Weiwei

[DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hello everyone, I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes. The merge payload