Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hi Sanjay, With respect to Ozone my two main concerns were: 1. Wether Ozone can help scaling out the namespace service in handling higher RPC workloads. I think we came to common conclusion that using Ozone as a block management layer is a reasonable path to scaling HDFS. The discussions are

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Konstantine Thanks for your feedback and comments over the last few months. Have we addressed all your issues and concerns? sanjay > On Feb 13, 2018, at 6:28 PM, sanjay Radia wrote: > > Sorry the formatting got messed by my email client. Here it is again > > >

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Sorry the formatting got messed by my email client. Here it is again Dear Hadoop Community Members, We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Dear Hadoop Community Members, We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments. The key questions raised were following How the new block storage

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hi Sanjay, Read your doc. I clearly see the value of Ozone with your use cases, but I agree with Stack and others the question why it should be a part of Hadoop isn't clear. More details in the jira:

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

> At a minimum, it should at least be using it’s own maven module for a > lot of the bits that generates it’s own maven jars so that we can split this > functionality up at build/test time. I expected this to be the case, but looks like it isn't. There's lot of value in splitting the

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hi folks! Thank you for sharing the design docs and the tremendous amount of work that has gone into Ozone. I'm grateful that atleast someone is trying to drastically improve HDFS. *If* there is a meeting to discuss this merge, could I please also be invited? Have we ever thought about

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Konstantine, Thanks for your comments, questions and feedback. I have attached a document to the HDFS-7240 jira that explains a design for scaling HDFS and how Ozone paves the way towards the full solution.

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

> On Nov 3, 2017, at 12:08 PM, Stack wrote: > > On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko > wrote: > >> It is an interesting question whether Ozone should be a part of Hadoop. > > I don't see a direct answer to this question. Is there one?

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko wrote: > Hey guys, > > It is an interesting question whether Ozone should be a part of Hadoop. > I don't see a direct answer to this question. Is there one? Pardon me if I've not seen it but I'm interested in the

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hi Konstantin, Thank you for taking out time to review ozone. I appreciate your comments and questions. > There are two main limitations in HDFS > a) The throughput of Namespace operations. Which is limited by the >number of RPCs the NameNode can handle > b) The number of objects (files +

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Hey guys, It is an interesting question whether Ozone should be a part of Hadoop. There are two main reasons why I think it should not. 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a sizable community behind, it looks to me like a whole new project. It is essentially a new

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Wow, big piece of work 1. Where is a PR/branch on github with rendered docs for us to look at? 2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details. thanks > On 19 Oct 2017, at 02:54, Yang Weiwei