Re: [VOTE] Merging branch HDFS-7240 to trunk

[Konstantin Shvachko]

The proposal to add it as a subproject of Hadoop makes sense to me. Thank
you Owen.
I am glad to have a path for scaling HDFS further, especially as it enters
areas like IoT and self-driving cars, where storage requirements are huge.

I am not very fond of the name HDSL, though. "Storage Layer" sounds too
generic.
May be something more descriptive, like HDDS / HDSS (Hadoop Dynamically
Distributed/Scaling Storage).
We can discuss this in the jira HDFS-10419.

Thanks,
--Konstantin

On Fri, Mar 16, 2018 at 3:17 PM, Sanjay Radia 
wrote:

> Owen,
> Thanks for your proposal.
> While I would have prefered to have HDSL in HDFS and also to be part
> of Hadoop releases  for the reasons stated earlier in this thread,
> I am willing to accept your proposal as a compromise to move this forward.
>
> Jitendra, Anu, Daryn, Andrew, Konstantine your thoughts?
>
> Thanks
>
> Sanjay
>
>
> On Mar 14, 2018, at 1:50 PM, Owen O'Malley > wrote:
>
> This discussion seems to have died down coming closer consensus without a
> resolution.
>
> I'd like to propose the following compromise:
>
> * HDSL become a subproject of Hadoop.
> * HDSL will release separately from Hadoop. Hadoop releases will not
> contain HDSL and vice versa.
> * HDSL will get its own jira instance so that the release tags stay
> separate.
> * On trunk (as opposed to release branches) HDSL will be a separate module
> in Hadoop's source tree. This will enable the HDSL to work on their trunk
> and the Hadoop trunk without making releases for every change.
> * Hadoop's trunk will only build HDSL if a non-default profile is enabled.
> * When Hadoop creates a release branch, the RM will delete the HDSL module
> from the branch.
> * HDSL will have their own Yetus checks and won't cause failures in the
> Hadoop patch check.
>
> I think this accomplishes most of the goals of encouraging HDSL development
> while minimizing the potential for disruption of HDFS development.
>
> Thoughts? Andrew, Jitendra, & Sanjay?
>
> Thanks,
>   Owen
>
>

Re: [VOTE] Merging branch HDFS-7240 to trunk

[sanjay Radia]

> On Mar 5, 2018, at 4:08 PM, Andrew Wang  wrote:
> 
> - NN on top HDSL where the NN uses the new block layer (Both Daryn and Owen 
> acknowledge the benefit of the new block layer).  We have two choices here
>  ** a) Evolve NN so that it can interact with both old and new block layer,
>  **  b) Fork and create new NN that works only with new block layer, the old 
> NN will continue to work with old block layer.
> There are trade-offs but clearly the 2nd option has least impact on the old 
> HDFS code.
> 
> Are you proposing that we pursue the 2nd option to integrate HDSL with HDFS?


Originally I would have prefered (a); but Owen made a strong case for (b) in my 
discussions with his last week.
Overall we need a broader discussion around the next steps for NN evolution and 
how to chart the course; I am not locked into any particular path or how we 
would do it. 
Let me make a more detailed response in HDFS-10419.

sanjay

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Owen O'Malley]

This discussion seems to have died down coming closer consensus without a
resolution.

I'd like to propose the following compromise:

* HDSL become a subproject of Hadoop.
* HDSL will release separately from Hadoop. Hadoop releases will not
contain HDSL and vice versa.
* HDSL will get its own jira instance so that the release tags stay
separate.
* On trunk (as opposed to release branches) HDSL will be a separate module
in Hadoop's source tree. This will enable the HDSL to work on their trunk
and the Hadoop trunk without making releases for every change.
* Hadoop's trunk will only build HDSL if a non-default profile is enabled.
* When Hadoop creates a release branch, the RM will delete the HDSL module
from the branch.
* HDSL will have their own Yetus checks and won't cause failures in the
Hadoop patch check.

I think this accomplishes most of the goals of encouraging HDSL development
while minimizing the potential for disruption of HDFS development.

Thoughts? Andrew, Jitendra, & Sanjay?

Thanks,
   Owen

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Owen O'Malley]

Hi Joep,

On Tue, Mar 6, 2018 at 6:50 PM, J. Rottinghuis 
wrote:

Obviously when people do want to use Ozone, then having it in the same repo
> is easier. The flipside is that, separate top-level project in the same
> repo or not, it adds to the Hadoop releases.
>

Apache projects are about the group of people who are working together.
There is a large overlap between the team working on HDFS and Ozone, which
is a lot of the motivation to keep project overhead to a minimum and not
start a new project.

Using the same releases or separate releases is a distinct choice. Many
Apache projects, such as Common and Maven, have multiple artifacts that
release independently. In Hive, we have two sub-projects that release
indepdendently: Hive Storage API, and Hive.

One thing we did during that split to minimize the challenges to the
developers was that Storage API and Hive have the same master branch.
However, since they have different releases, they have their own release
branches and release numbers.

If there is a change in Ozone and a new release needed, it would have to
> wait for a Hadoop release. Ditto if there is a Hadoop release and there is
> an issue with Ozone. The case that one could turn off Ozone through a Maven
> profile works only to some extend.
> If we have done a 3.x release with Ozone in it, would it make sense to do
> a 3.y release with y>x without Ozone in it? That would be weird.
>

Actually, if Ozone is marked as unstable/evolving (we should actually have
an even stronger warning for a feature preview), we could remove it in a
3.x. If a user picks up a feature before it is stable, we try to provide a
stable platform, but mistakes happen. Introducing an incompatible change to
the Ozone API between 3.1 and 3.2 wouldn't be good, but it wouldn't be the
end of the world.

.. Owen

Re: [VOTE] Merging branch HDFS-7240 to trunk

[sanjay Radia]

sier. The flipside is that, separate top-level project in the same
> repo or not, it adds to the Hadoop releases. If there is a change in Ozone
> and a new release needed, it would have to wait for a Hadoop release. Ditto
> if there is a Hadoop release and there is an issue with Ozone. The case
> that one could turn off Ozone through a Maven profile works only to some
> extend.
> If we have done a 3.x release with Ozone in it, would it make sense to do a
> 3.y release with y>x without Ozone in it? That would be weird.
> 
> This does sound like a Hadoop 4 feature. Compatibility with lots of new
> features in Hadoop 3 need to be worked out. We're still working on jumping
> to a Hadoop 2.9 release and then working on getting a step-store release to
> 3.0 to bridge compatibility issues. I'm afraid that adding a very large new
> feature into trunk now, essentially makes going to Hadoop 3 not viable for
> quite a while. That would be a bummer for all the feature work that has
> gone into Hadoop 3. Encryption and erasure encoding are very appealing
> features, especially in light of meeting GDPR requirements.
> 
> I'd argue to pull out those pieces that make sense in Hadoop 3, merge those
> in and keep the rest in a separate project. Iterate quickly in that
> separate project, you can have a separate set of committers, you can do
> separate release cycle. If that develops Ozone into _the_ new block layer
> for all use cases (even when people want to give up on encryption, erasure
> encoding, or feature parity is reached) then we can jump of that bridge
> when we reach it. I think adding a very large chunk of code that relatively
> few people in the community are familiar with isn't necessarily going to
> help Hadoop at this time.
> 
> Cheers,
> 
> Joep
> 
> On Tue, Mar 6, 2018 at 2:32 PM, Jitendra Pandey <jiten...@hortonworks.com>
> wrote:
> 
>> Hi Andrew,
>> 
>> I think we can eliminate the maintenance costs even in the same repo. We
>> can make following changes that incorporate suggestions from Daryn and Owen
>> as well.
>> 1. Hadoop-hdsl-project will be at the root of hadoop repo, in a separate
>> directory.
>> 2. There will be no dependencies from common, yarn and hdfs to hdsl/ozone.
>> 3. Based on Daryn’s suggestion, the Hdsl can be optionally (via config) be
>> loaded in DN as a pluggable module.
>> If not loaded, there will be absolutely no code path through hdsl or
>> ozone.
>> 4. To further make it easier for folks building hadoop, we can support a
>> maven profile for hdsl/ozone. If the profile is deactivated hdsl/ozone will
>> not be built.
>> For example, Cloudera can choose to skip even compiling/building
>> hdsl/ozone and therefore no maintenance overhead whatsoever.
>> HADOOP-14453 has a patch that shows how it can be done.
>> 
>> Arguably, there are two kinds of maintenance costs. Costs for developers
>> and the cost for users.
>> - Developers: A maven profile as noted in point (3) and (4) above
>> completely addresses the concern for developers
>> as there are no compile time dependencies
>> and further, they can choose not to build ozone/hdsl.
>> - User: Cost to users will be completely alleviated if ozone/hdsl is not
>> loaded as mentioned in point (3) above.
>> 
>> jitendra
>> 
>> From: Andrew Wang <andrew.w...@cloudera.com>
>> Date: Monday, March 5, 2018 at 3:54 PM
>> To: Wangda Tan <wheele...@gmail.com>
>> Cc: Owen O'Malley <owen.omal...@gmail.com>, Daryn Sharp
>> <da...@oath.com.invalid>, Jitendra Pandey <jiten...@hortonworks.com>,
>> hdfs-dev <hdfs-...@hadoop.apache.org>, "common-...@hadoop.apache.org" <
>> common-...@hadoop.apache.org>, "yarn-...@hadoop.apache.org" <
>> yarn-...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <
>> mapreduce-dev@hadoop.apache.org>
>> Subject: Re: [VOTE] Merging branch HDFS-7240 to trunk
>> 
>> Hi Owen, Wangda,
>> 
>> Thanks for clearly laying out the subproject options, that helps the
>> discussion.
>> 
>> I'm all onboard with the idea of regular releases, and it's something I
>> tried to do with the 3.0 alphas and betas. The problem though isn't a lack
>> of commitment from feature developers like Sanjay or Jitendra; far from it!
>> I think every feature developer makes a reasonable effort to test their
>> code before it's merged. Yet, my experience as an RM is that more code
>> comes with more risk. I don't believe that Ozone is special or different in
>> this regard. It comes with a maintenance 

Re: [VOTE] Merging branch HDFS-7240 to trunk

[J. Rottinghuis]

ey <owen.omal...@gmail.com>, Daryn Sharp
> <da...@oath.com.invalid>, Jitendra Pandey <jiten...@hortonworks.com>,
> hdfs-dev <hdfs-...@hadoop.apache.org>, "common-...@hadoop.apache.org" <
> common-...@hadoop.apache.org>, "yarn-...@hadoop.apache.org" <
> yarn-...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <
> mapreduce-dev@hadoop.apache.org>
> Subject: Re: [VOTE] Merging branch HDFS-7240 to trunk
>
> Hi Owen, Wangda,
>
> Thanks for clearly laying out the subproject options, that helps the
> discussion.
>
> I'm all onboard with the idea of regular releases, and it's something I
> tried to do with the 3.0 alphas and betas. The problem though isn't a lack
> of commitment from feature developers like Sanjay or Jitendra; far from it!
> I think every feature developer makes a reasonable effort to test their
> code before it's merged. Yet, my experience as an RM is that more code
> comes with more risk. I don't believe that Ozone is special or different in
> this regard. It comes with a maintenance cost, not a maintenance benefit.
>
>
> I'm advocating for #3: separate source, separate release. Since HDSL
> stability and FSN/BM refactoring are still a ways out, I don't want to
> incur a maintenance cost now. I sympathize with the sentiment that working
> cross-repo is harder than within same repo, but the right tooling can make
> this a lot easier (e.g. git submodule, Google's repo tool). We have
> experience doing this internally here at Cloudera, and I'm happy to share
> knowledge and possibly code.
>
> Best,
> Andrew
>
> On Fri, Mar 2, 2018 at 4:41 PM, Wangda Tan <wheele...@gmail.com> wrote:
> I like the idea of same source / same release and put Ozone's source under
> a different directory.
>
> Like Owen mentioned, It gonna be important for all parties to keep a
> regular and shorter release cycle for Hadoop, e.g. 3-4 months between minor
> releases. Users can try features and give feedbacks to stabilize feature
> earlier; developers can be happier since efforts will be consumed by users
> soon after features get merged. In addition to this, if features merged to
> trunk after reasonable tests/review, Andrew's concern may not be a problem
> anymore:
>
> bq. Finally, I earnestly believe that Ozone/HDSL itself would benefit from
> being a separate project. Ozone could release faster and iterate more
> quickly if it wasn't hampered by Hadoop's release schedule and security and
> compatibility requirements.
>
> Thanks,
> Wangda
>
>
> On Fri, Mar 2, 2018 at 4:24 PM, Owen O'Malley <owen.omal...@gmail.com>
> wrote:
> On Thu, Mar 1, 2018 at 11:03 PM, Andrew Wang <andrew.w...@cloudera.com>
> wrote:
>
> Owen mentioned making a Hadoop subproject; we'd have to
> > hash out what exactly this means (I assume a separate repo still managed
> by
> > the Hadoop project), but I think we could make this work if it's more
> > attractive than incubation or a new TLP.
>
>
> Ok, there are multiple levels of sub-projects that all make sense:
>
>- Same source tree, same releases - examples like HDFS & YARN
>- Same master branch, separate releases and release branches - Hive's
>Storage API vs Hive. It is in the source tree for the master branch, but
>has distinct releases and release branches.
>- Separate source, separate release - Apache Commons.
>
> There are advantages and disadvantages to each. I'd propose that we use the
> same source, same release pattern for Ozone. Note that we tried and later
> reverted doing Common, HDFS, and YARN as separate source, separate release
> because it was too much trouble. I like Daryn's idea of putting it as a top
> level directory in Hadoop and making sure that nothing in Common, HDFS, or
> YARN depend on it. That way if a Release Manager doesn't think it is ready
> for release, it can be trivially removed before the release.
>
> One thing about using the same releases, Sanjay and Jitendra are signing up
> to make much more regular bugfix and minor releases in the near future. For
> example, they'll need to make 3.2 relatively soon to get it released and
> then 3.3 somewhere in the next 3 to 6 months. That would be good for the
> project. Hadoop needs more regular releases and fewer big bang releases.
>
> .. Owen
>
>
>
>
>

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Jitendra Pandey]

Hi Andrew, 
  
 I think we can eliminate the maintenance costs even in the same repo. We can 
make following changes that incorporate suggestions from Daryn and Owen as well.
1. Hadoop-hdsl-project will be at the root of hadoop repo, in a separate 
directory.
2. There will be no dependencies from common, yarn and hdfs to hdsl/ozone.
3. Based on Daryn’s suggestion, the Hdsl can be optionally (via config) be 
loaded in DN as a pluggable module. 
 If not loaded, there will be absolutely no code path through hdsl or ozone.
4. To further make it easier for folks building hadoop, we can support a maven 
profile for hdsl/ozone. If the profile is deactivated hdsl/ozone will not be 
built.
 For example, Cloudera can choose to skip even compiling/building 
hdsl/ozone and therefore no maintenance overhead whatsoever.
 HADOOP-14453 has a patch that shows how it can be done.

Arguably, there are two kinds of maintenance costs. Costs for developers and 
the cost for users.
- Developers: A maven profile as noted in point (3) and (4) above completely 
addresses the concern for developers 
 as there are no compile time dependencies and 
further, they can choose not to build ozone/hdsl.
- User: Cost to users will be completely alleviated if ozone/hdsl is not loaded 
as mentioned in point (3) above.

jitendra

From: Andrew Wang <andrew.w...@cloudera.com>
Date: Monday, March 5, 2018 at 3:54 PM
To: Wangda Tan <wheele...@gmail.com>
Cc: Owen O'Malley <owen.omal...@gmail.com>, Daryn Sharp 
<da...@oath.com.invalid>, Jitendra Pandey <jiten...@hortonworks.com>, hdfs-dev 
<hdfs-...@hadoop.apache.org>, "common-...@hadoop.apache.org" 
<common-...@hadoop.apache.org>, "yarn-...@hadoop.apache.org" 
<yarn-...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" 
<mapreduce-dev@hadoop.apache.org>
Subject: Re: [VOTE] Merging branch HDFS-7240 to trunk

Hi Owen, Wangda, 

Thanks for clearly laying out the subproject options, that helps the discussion.

I'm all onboard with the idea of regular releases, and it's something I tried 
to do with the 3.0 alphas and betas. The problem though isn't a lack of 
commitment from feature developers like Sanjay or Jitendra; far from it! I 
think every feature developer makes a reasonable effort to test their code 
before it's merged. Yet, my experience as an RM is that more code comes with 
more risk. I don't believe that Ozone is special or different in this regard. 
It comes with a maintenance cost, not a maintenance benefit.


I'm advocating for #3: separate source, separate release. Since HDSL stability 
and FSN/BM refactoring are still a ways out, I don't want to incur a 
maintenance cost now. I sympathize with the sentiment that working cross-repo 
is harder than within same repo, but the right tooling can make this a lot 
easier (e.g. git submodule, Google's repo tool). We have experience doing this 
internally here at Cloudera, and I'm happy to share knowledge and possibly code.

Best,
Andrew

On Fri, Mar 2, 2018 at 4:41 PM, Wangda Tan <wheele...@gmail.com> wrote:
I like the idea of same source / same release and put Ozone's source under a 
different directory. 

Like Owen mentioned, It gonna be important for all parties to keep a regular 
and shorter release cycle for Hadoop, e.g. 3-4 months between minor releases. 
Users can try features and give feedbacks to stabilize feature earlier; 
developers can be happier since efforts will be consumed by users soon after 
features get merged. In addition to this, if features merged to trunk after 
reasonable tests/review, Andrew's concern may not be a problem anymore: 

bq. Finally, I earnestly believe that Ozone/HDSL itself would benefit from
being a separate project. Ozone could release faster and iterate more
quickly if it wasn't hampered by Hadoop's release schedule and security and
compatibility requirements.

Thanks,
Wangda


On Fri, Mar 2, 2018 at 4:24 PM, Owen O'Malley <owen.omal...@gmail.com> wrote:
On Thu, Mar 1, 2018 at 11:03 PM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

Owen mentioned making a Hadoop subproject; we'd have to
> hash out what exactly this means (I assume a separate repo still managed by
> the Hadoop project), but I think we could make this work if it's more
> attractive than incubation or a new TLP.


Ok, there are multiple levels of sub-projects that all make sense:

   - Same source tree, same releases - examples like HDFS & YARN
   - Same master branch, separate releases and release branches - Hive's
   Storage API vs Hive. It is in the source tree for the master branch, but
   has distinct releases and release branches.
   - Separate source, separate release - Apache Commons.

There are advantages and disadvantages to each. I'd propose that we use the
same source, same release pattern for Ozone. Note that we tried and later
reverted doing Common, HDFS, and YA

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Andrew Wang]

Hi Sanjay, thanks for the response, replying inline:

- NN on top HDSL where the NN uses the new block layer (Both Daryn and Owen
> acknowledge the benefit of the new block layer).  We have two choices here
>  ** a) Evolve NN so that it can interact with both old and new block layer,
>  **  b) Fork and create new NN that works only with new block layer, the
> old NN will continue to work with old block layer.
> There are trade-offs but clearly the 2nd option has least impact on the
> old HDFS code.
>
> Are you proposing that we pursue the 2nd option to integrate HDSL with
HDFS?


> - Share the HDSL’s netty  protocol engine with HDFS block layer.  After
> HDSL and Ozone has stabilized the engine, put the new netty engine in
> either HDFS or in Hadoop common - HDSL will use it from there. The HDFS
> community  has been talking about moving to better thread model for HDFS
> DNs since release 0.16!!
>
> The Netty-based protocol engine seems like it could be contributed
separately from HDSL. I'd be interested to learn more about the performance
and other improvements from this new engine.


> - Shallow copy. Here HDSL needs a way to get the actual linux file system
> links - HDFS block layer needs  to provide a private secure API to get file
> names of blocks so that HDSL can do a hard link (hence shallow copy)o
>

Why isn't this possible with two processes? SCR for instance securely
passes file descriptors between the DN and client over a unix domain
socket. I'm sure we can construct a protocol that securely and efficiently
creates hardlinks.

It also sounds like this shallow copy won't work with features like HDFS
encryption or erasure coding, which diminishes its utility. We also don't
even have HDFS-to-HDFS shallow copy yet, so HDFS-to-Ozone shallow copy is
even further out.

Best,
Andrew

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Andrew Wang]

Hi Owen, Wangda,

Thanks for clearly laying out the subproject options, that helps the
discussion.

I'm all onboard with the idea of regular releases, and it's something I
tried to do with the 3.0 alphas and betas. The problem though isn't a lack
of commitment from feature developers like Sanjay or Jitendra; far from it!
I think every feature developer makes a reasonable effort to test their
code before it's merged. Yet, my experience as an RM is that more code
comes with more risk. I don't believe that Ozone is special or different in
this regard. It comes with a maintenance cost, not a maintenance benefit.

I'm advocating for #3: separate source, separate release. Since HDSL
stability and FSN/BM refactoring are still a ways out, I don't want to
incur a maintenance cost now. I sympathize with the sentiment that working
cross-repo is harder than within same repo, but the right tooling can make
this a lot easier (e.g. git submodule, Google's repo tool). We have
experience doing this internally here at Cloudera, and I'm happy to share
knowledge and possibly code.

Best,
Andrew

On Fri, Mar 2, 2018 at 4:41 PM, Wangda Tan  wrote:

> I like the idea of same source / same release and put Ozone's source under
> a different directory.
>
> Like Owen mentioned, It gonna be important for all parties to keep a
> regular and shorter release cycle for Hadoop, e.g. 3-4 months between minor
> releases. Users can try features and give feedbacks to stabilize feature
> earlier; developers can be happier since efforts will be consumed by users
> soon after features get merged. In addition to this, if features merged to
> trunk after reasonable tests/review, Andrew's concern may not be a problem
> anymore:
>
> bq. Finally, I earnestly believe that Ozone/HDSL itself would benefit from
> being a separate project. Ozone could release faster and iterate more
> quickly if it wasn't hampered by Hadoop's release schedule and security and
> compatibility requirements.
>
> Thanks,
> Wangda
>
>
> On Fri, Mar 2, 2018 at 4:24 PM, Owen O'Malley 
> wrote:
>
>> On Thu, Mar 1, 2018 at 11:03 PM, Andrew Wang 
>> wrote:
>>
>> Owen mentioned making a Hadoop subproject; we'd have to
>> > hash out what exactly this means (I assume a separate repo still
>> managed by
>> > the Hadoop project), but I think we could make this work if it's more
>> > attractive than incubation or a new TLP.
>>
>>
>> Ok, there are multiple levels of sub-projects that all make sense:
>>
>>- Same source tree, same releases - examples like HDFS & YARN
>>- Same master branch, separate releases and release branches - Hive's
>>Storage API vs Hive. It is in the source tree for the master branch,
>> but
>>has distinct releases and release branches.
>>- Separate source, separate release - Apache Commons.
>>
>> There are advantages and disadvantages to each. I'd propose that we use
>> the
>> same source, same release pattern for Ozone. Note that we tried and later
>> reverted doing Common, HDFS, and YARN as separate source, separate release
>> because it was too much trouble. I like Daryn's idea of putting it as a
>> top
>> level directory in Hadoop and making sure that nothing in Common, HDFS, or
>> YARN depend on it. That way if a Release Manager doesn't think it is ready
>> for release, it can be trivially removed before the release.
>>
>> One thing about using the same releases, Sanjay and Jitendra are signing
>> up
>> to make much more regular bugfix and minor releases in the near future.
>> For
>> example, they'll need to make 3.2 relatively soon to get it released and
>> then 3.3 somewhere in the next 3 to 6 months. That would be good for the
>> project. Hadoop needs more regular releases and fewer big bang releases.
>>
>> .. Owen
>>
>
>

Re: [VOTE] Merging branch HDFS-7240 to trunk

[sanjay Radia]

Andrew
  Thanks for your response. 

 In this email let me focus on maintenance and unnecessary impact on HDFS.
Daryn also touched on this topic and looked at the code base from the developer 
impact point of view. He appreciated that the code is separate and I agree with 
his suggestion to move it further up the src tree (e.g. Hadoop-hdsl-project or 
hadoop-hdfs-project/hadoop-hdsl). He also gave a good analogy to the store: do 
not break things as you change and evolve the store. Let’s look at the areas of 
future interaction as examples.

- NN on top HDSL where the NN uses the new block layer (Both Daryn and Owen 
acknowledge the benefit of the new block layer).  We have two choices here 
 ** a) Evolve NN so that it can interact with both old and new block layer, 
 **  b) Fork and create new NN that works only with new block layer, the old NN 
will continue to work with old block layer. 
There are trade-offs but clearly the 2nd option has least impact on the old 
HDFS code.  

- Share the HDSL’s netty  protocol engine with HDFS block layer.  After HDSL 
and Ozone has stabilized the engine, put the new netty engine in either HDFS or 
in Hadoop common - HDSL will use it from there. The HDFS community  has been 
talking about moving to better thread model for HDFS DNs since release 0.16!!

- Shallow copy. Here HDSL needs a way to get the actual linux file system links 
- HDFS block layer needs  to provide a private secure API to get file names of 
blocks so that HDSL can do a hard link (hence shallow copy)o

The first 2 examples are beneficial to existing HDFS and the maintenance burden 
can be minimized and worth the benefits (2x NN scalability!! And more efficient 
protocol engine). The 3rd is only beneficial to HDFS users who want the 
scalability of the new HDSL/Ozone code in a side-by-side system; here the cost 
is providing a  private API to access the block file name. 


sanjay

> On Mar 1, 2018, at 11:03 PM, Andrew Wang  wrote:
> 
> Hi Sanjay,
> 
> I have different opinions about what's important and how to eventually
> integrate this code, and that's not because I'm "conveniently ignoring"
> your responses. I'm also not making some of the arguments you claim I am
> making. Attacking arguments I'm not making is not going to change my mind,
> so let's bring it back to the arguments I am making.
> 
> Here's what it comes down to: HDFS-on-HDSL is not going to be ready in the
> near-term, and it comes with a maintenance cost.
> 
> I did read the proposal on HDFS-10419 and I understood that HDFS-on-HDSL
> integration does not necessarily require a lock split. However, there still
> needs to be refactoring to clearly define the FSN and BM interfaces and
> make the BM pluggable so HDSL can be swapped in. This is a major
> undertaking and risky. We did a similar refactoring in 2.x which made
> backports hard and introduced bugs. I don't think we should have done this
> in a minor release.
> 
> Furthermore, I don't know what your expectation is on how long it will take
> to stabilize HDSL, but this horizon for other storage systems is typically
> measured in years rather than months.
> 
> Both of these feel like Hadoop 4 items: a ways out yet.
> 
> Moving on, there is a non-trivial maintenance cost to having this new code
> in the code base. Ozone bugs become our bugs. Ozone dependencies become our
> dependencies. Ozone's security flaws are our security flaws. All of this
> negatively affects our already lumbering release schedule, and thus our
> ability to deliver and iterate on the features we're already trying to
> ship. Even if Ozone is separate and off by default, this is still a large
> amount of code that comes with a large maintenance cost. I don't want to
> incur this cost when the benefit is still a ways out.
> 
> We disagree on the necessity of sharing a repo and sharing operational
> behaviors. Libraries exist as a method for sharing code. HDFS also hardly
> has a monopoly on intermediating storage today. Disks are shared with MR
> shuffle, Spark/Impala spill, log output, Kudu, Kafka, etc. Operationally
> we've made this work. Having Ozone/HDSL in a separate process can even be
> seen as an operational advantage since it's isolated. I firmly believe that
> we can solve any implementation issues even with separate processes.
> 
> This is why I asked about making this a separate project. Given that these
> two efforts (HDSL stabilization and NN refactoring) are a ways out, the
> best way to get Ozone/HDSL in the hands of users today is to release it as
> its own project. Owen mentioned making a Hadoop subproject; we'd have to
> hash out what exactly this means (I assume a separate repo still managed by
> the Hadoop project), but I think we could make this work if it's more
> attractive than incubation or a new TLP.
> 
> I'm excited about the possibilities of both HDSL and the NN refactoring in
> ensuring a future for HDFS for years to come. A pluggable block manager
> 

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Anu Engineer]

Hi Owen,

  >> 1. It is hard to tell what has changed. git rebase -i tells me the
  >> branch has 722 commits. The rebase failed with a conflict. It would really
   >> help if you rebased to current trunk.

Thanks for the comments. I have merged trunk to HDFS-7240 branch. 
Hopefully, this makes it easy to look at the changes; I have committed the 
change required to fix the conflict as a separate commit to make it easy for 
you to see.

Thanks
Anu


On 3/2/18, 4:42 PM, "Wangda Tan"  wrote:

I like the idea of same source / same release and put Ozone's source under
a different directory.

Like Owen mentioned, It gonna be important for all parties to keep a
regular and shorter release cycle for Hadoop, e.g. 3-4 months between minor
releases. Users can try features and give feedbacks to stabilize feature
earlier; developers can be happier since efforts will be consumed by users
soon after features get merged. In addition to this, if features merged to
trunk after reasonable tests/review, Andrew's concern may not be a problem
anymore:

bq. Finally, I earnestly believe that Ozone/HDSL itself would benefit from
being a separate project. Ozone could release faster and iterate more
quickly if it wasn't hampered by Hadoop's release schedule and security and
compatibility requirements.

Thanks,
Wangda


On Fri, Mar 2, 2018 at 4:24 PM, Owen O'Malley 
wrote:

> On Thu, Mar 1, 2018 at 11:03 PM, Andrew Wang 
> wrote:
>
> Owen mentioned making a Hadoop subproject; we'd have to
> > hash out what exactly this means (I assume a separate repo still managed
> by
> > the Hadoop project), but I think we could make this work if it's more
> > attractive than incubation or a new TLP.
>
>
> Ok, there are multiple levels of sub-projects that all make sense:
>
>- Same source tree, same releases - examples like HDFS & YARN
>- Same master branch, separate releases and release branches - Hive's
>Storage API vs Hive. It is in the source tree for the master branch, 
but
>has distinct releases and release branches.
>- Separate source, separate release - Apache Commons.
>
> There are advantages and disadvantages to each. I'd propose that we use 
the
> same source, same release pattern for Ozone. Note that we tried and later
> reverted doing Common, HDFS, and YARN as separate source, separate release
> because it was too much trouble. I like Daryn's idea of putting it as a 
top
> level directory in Hadoop and making sure that nothing in Common, HDFS, or
> YARN depend on it. That way if a Release Manager doesn't think it is ready
> for release, it can be trivially removed before the release.
>
> One thing about using the same releases, Sanjay and Jitendra are signing 
up
> to make much more regular bugfix and minor releases in the near future. 
For
> example, they'll need to make 3.2 relatively soon to get it released and
> then 3.3 somewhere in the next 3 to 6 months. That would be good for the
> project. Hadoop needs more regular releases and fewer big bang releases.
>
> .. Owen
>

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Wangda Tan]

I like the idea of same source / same release and put Ozone's source under
a different directory.

Like Owen mentioned, It gonna be important for all parties to keep a
regular and shorter release cycle for Hadoop, e.g. 3-4 months between minor
releases. Users can try features and give feedbacks to stabilize feature
earlier; developers can be happier since efforts will be consumed by users
soon after features get merged. In addition to this, if features merged to
trunk after reasonable tests/review, Andrew's concern may not be a problem
anymore:

bq. Finally, I earnestly believe that Ozone/HDSL itself would benefit from
being a separate project. Ozone could release faster and iterate more
quickly if it wasn't hampered by Hadoop's release schedule and security and
compatibility requirements.

Thanks,
Wangda


On Fri, Mar 2, 2018 at 4:24 PM, Owen O'Malley 
wrote:

> On Thu, Mar 1, 2018 at 11:03 PM, Andrew Wang 
> wrote:
>
> Owen mentioned making a Hadoop subproject; we'd have to
> > hash out what exactly this means (I assume a separate repo still managed
> by
> > the Hadoop project), but I think we could make this work if it's more
> > attractive than incubation or a new TLP.
>
>
> Ok, there are multiple levels of sub-projects that all make sense:
>
>- Same source tree, same releases - examples like HDFS & YARN
>- Same master branch, separate releases and release branches - Hive's
>Storage API vs Hive. It is in the source tree for the master branch, but
>has distinct releases and release branches.
>- Separate source, separate release - Apache Commons.
>
> There are advantages and disadvantages to each. I'd propose that we use the
> same source, same release pattern for Ozone. Note that we tried and later
> reverted doing Common, HDFS, and YARN as separate source, separate release
> because it was too much trouble. I like Daryn's idea of putting it as a top
> level directory in Hadoop and making sure that nothing in Common, HDFS, or
> YARN depend on it. That way if a Release Manager doesn't think it is ready
> for release, it can be trivially removed before the release.
>
> One thing about using the same releases, Sanjay and Jitendra are signing up
> to make much more regular bugfix and minor releases in the near future. For
> example, they'll need to make 3.2 relatively soon to get it released and
> then 3.3 somewhere in the next 3 to 6 months. That would be good for the
> project. Hadoop needs more regular releases and fewer big bang releases.
>
> .. Owen
>

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Owen O'Malley]

On Thu, Mar 1, 2018 at 11:03 PM, Andrew Wang 
wrote:

Owen mentioned making a Hadoop subproject; we'd have to
> hash out what exactly this means (I assume a separate repo still managed by
> the Hadoop project), but I think we could make this work if it's more
> attractive than incubation or a new TLP.


Ok, there are multiple levels of sub-projects that all make sense:

   - Same source tree, same releases - examples like HDFS & YARN
   - Same master branch, separate releases and release branches - Hive's
   Storage API vs Hive. It is in the source tree for the master branch, but
   has distinct releases and release branches.
   - Separate source, separate release - Apache Commons.

There are advantages and disadvantages to each. I'd propose that we use the
same source, same release pattern for Ozone. Note that we tried and later
reverted doing Common, HDFS, and YARN as separate source, separate release
because it was too much trouble. I like Daryn's idea of putting it as a top
level directory in Hadoop and making sure that nothing in Common, HDFS, or
YARN depend on it. That way if a Release Manager doesn't think it is ready
for release, it can be trivially removed before the release.

One thing about using the same releases, Sanjay and Jitendra are signing up
to make much more regular bugfix and minor releases in the near future. For
example, they'll need to make 3.2 relatively soon to get it released and
then 3.3 somewhere in the next 3 to 6 months. That would be good for the
project. Hadoop needs more regular releases and fewer big bang releases.

.. Owen

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Andrew Wang]

Hi Sanjay,

I have different opinions about what's important and how to eventually
integrate this code, and that's not because I'm "conveniently ignoring"
your responses. I'm also not making some of the arguments you claim I am
making. Attacking arguments I'm not making is not going to change my mind,
so let's bring it back to the arguments I am making.

Here's what it comes down to: HDFS-on-HDSL is not going to be ready in the
near-term, and it comes with a maintenance cost.

I did read the proposal on HDFS-10419 and I understood that HDFS-on-HDSL
integration does not necessarily require a lock split. However, there still
needs to be refactoring to clearly define the FSN and BM interfaces and
make the BM pluggable so HDSL can be swapped in. This is a major
undertaking and risky. We did a similar refactoring in 2.x which made
backports hard and introduced bugs. I don't think we should have done this
in a minor release.

Furthermore, I don't know what your expectation is on how long it will take
to stabilize HDSL, but this horizon for other storage systems is typically
measured in years rather than months.

Both of these feel like Hadoop 4 items: a ways out yet.

Moving on, there is a non-trivial maintenance cost to having this new code
in the code base. Ozone bugs become our bugs. Ozone dependencies become our
dependencies. Ozone's security flaws are our security flaws. All of this
negatively affects our already lumbering release schedule, and thus our
ability to deliver and iterate on the features we're already trying to
ship. Even if Ozone is separate and off by default, this is still a large
amount of code that comes with a large maintenance cost. I don't want to
incur this cost when the benefit is still a ways out.

We disagree on the necessity of sharing a repo and sharing operational
behaviors. Libraries exist as a method for sharing code. HDFS also hardly
has a monopoly on intermediating storage today. Disks are shared with MR
shuffle, Spark/Impala spill, log output, Kudu, Kafka, etc. Operationally
we've made this work. Having Ozone/HDSL in a separate process can even be
seen as an operational advantage since it's isolated. I firmly believe that
we can solve any implementation issues even with separate processes.

This is why I asked about making this a separate project. Given that these
two efforts (HDSL stabilization and NN refactoring) are a ways out, the
best way to get Ozone/HDSL in the hands of users today is to release it as
its own project. Owen mentioned making a Hadoop subproject; we'd have to
hash out what exactly this means (I assume a separate repo still managed by
the Hadoop project), but I think we could make this work if it's more
attractive than incubation or a new TLP.

I'm excited about the possibilities of both HDSL and the NN refactoring in
ensuring a future for HDFS for years to come. A pluggable block manager
would also let us experiment with things like HDFS-on-S3, increasingly
important in a cloud-centric world. CBlock would bring HDFS to new usecases
around generic container workloads. However, given the timeline for
completing these efforts, now is not the time to merge.

Best,
Andrew

On Thu, Mar 1, 2018 at 5:33 PM, Daryn Sharp  wrote:

> I’m generally neutral and looked foremost at developer impact.  Ie.  Will
> it be so intertwined with hdfs that each project risks destabilizing the
> other?  Will developers with no expertise in ozone will be impeded?  I
> think the answer is currently no.  These are the intersections and some
> concerns based on the assumption ozone is accepted into the project:
>
>
> Common
>
> Appear to be a number of superfluous changes.  The conf servlet must not be
> polluted with specific references and logic for ozone.  We don’t create
> dependencies from common to hdfs, mapred, yarn, hive, etc.  Common must be
> “ozone free”.
>
>
> Datanode
>
> I expected ozone changes to be intricately linked with the existing blocks
> map, dataset, volume, etc.  Thankfully it’s not.  As an independent
> service, the DN should not be polluted with specific references to ozone.
> If ozone is in the project, the DN should have a generic plugin interface
> conceptually similar to the NM aux services.
>
>
> Namenode
>
> No impact, currently, but certainly will be…
>
>
> Code Location
>
> I don’t feel hadoop-hdfs-project/hadoop-hdfs is an acceptable location.
> I’d rather see hadoop-hdfs-project/hadoop-hdsl, or even better
> hadoop-hdsl-project.  This clean separation will make it easier to later
> spin off or pull in depending on which way we vote.
>
>
> Dependencies
>
> Owen hit upon his before I could send.  Hadoop is already bursting with
> dependencies, I hope this doesn’t pull in a lot more.
>
>
> ––
>
>
> Do I think ozone be should be a separate project?  If we view it only as a
> competing filesystem, then clearly yes.  If it’s a low risk evolutionary
> step with near-term benefits, no, we want to keep it close and help it
> evolve. 

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Daryn Sharp]

I’m generally neutral and looked foremost at developer impact.  Ie.  Will
it be so intertwined with hdfs that each project risks destabilizing the
other?  Will developers with no expertise in ozone will be impeded?  I
think the answer is currently no.  These are the intersections and some
concerns based on the assumption ozone is accepted into the project:


Common

Appear to be a number of superfluous changes.  The conf servlet must not be
polluted with specific references and logic for ozone.  We don’t create
dependencies from common to hdfs, mapred, yarn, hive, etc.  Common must be
“ozone free”.


Datanode

I expected ozone changes to be intricately linked with the existing blocks
map, dataset, volume, etc.  Thankfully it’s not.  As an independent
service, the DN should not be polluted with specific references to ozone.
If ozone is in the project, the DN should have a generic plugin interface
conceptually similar to the NM aux services.


Namenode

No impact, currently, but certainly will be…


Code Location

I don’t feel hadoop-hdfs-project/hadoop-hdfs is an acceptable location.
I’d rather see hadoop-hdfs-project/hadoop-hdsl, or even better
hadoop-hdsl-project.  This clean separation will make it easier to later
spin off or pull in depending on which way we vote.


Dependencies

Owen hit upon his before I could send.  Hadoop is already bursting with
dependencies, I hope this doesn’t pull in a lot more.


––


Do I think ozone be should be a separate project?  If we view it only as a
competing filesystem, then clearly yes.  If it’s a low risk evolutionary
step with near-term benefits, no, we want to keep it close and help it
evolve.  I think ozone/hdsl/whatever has been poorly marketed and an
umbrella term for too many technologies that should perhaps be split.  I'm
interested in the container block management.  I have little interest at
this time in the key store.


The usability of ozone, specifically container management, is unclear to
me.  It lacks basic features like changing replication factors, append, a
migration path, security, etc - I know there are good plans for all of it -
yet another goal is splicing into the NN.  That’s a lot of high priority
items to tackle that need to be carefully orchestrated before contemplating
BM replacement.  Each of those is a non-starter for (my) production
environment.  We need to make sure we can reach a consensus on the block
level functionality before rushing it into the NN.  That’s independent of
whether allowing it into the project.


The BM/SCM changes to the NN are realistically going to be contentious &
destabilizing.  If done correctly, the BM separation will be a big win for
the NN.  If ozone is out, by necessity interfaces will need to be stable
and well-defined but we won’t get that right for a long time.  Interface
and logic changes that break the other will be difficult to coordinate and
we’ll likely veto changes that impact the other.  If ozone is in, we can
hopefully synchronize the changes with less friction, but it greatly
increases the chances of developers riddling the NN with hacks and/or ozone
specific logic that makes it even more brittle.  I will note we need to be
vigilant against pervasive conditionals (ie. EC, snapshots).


In either case, I think ozone must agree to not impede current hdfs work.
I’ll compare to hdfs is a store owner that plans to maybe retire in 5
years.  A potential new owner (ozone) is lined up and hdfs graciously gives
them no-rent space (the DN).  Precondition is help improve the store.
Don’t make a mess and expect hdfs to clean it up.  Don’t make renovations
that complicate hdfs but ignore it due to anticipation of its
departure/demise.  I’m not implying that’s currently happening, it’s just
what I don’t want to see.


We as a community and our customers need an evolution, not a revolution,
and definitively not a civil war.  Hdfs has too much legacy code rot that
is hard to change.  Too many poorly implemented features.   Perhaps I’m
overly optimistic that freshly redesigned code can counterbalance
performance degradations in the NN.  I’m also reluctant, but realize it is
being driven by some hdfs veterans that know/understand historical hdfs
design strengths and flaws.


If the initially cited issues are addressed, I’m +0.5 for the concept of
bringing in ozone if it's not going to be a proverbial bull in the china
shop.


Daryn

On Mon, Feb 26, 2018 at 3:18 PM, Jitendra Pandey 
wrote:

> Dear folks,
>We would like to start a vote to merge HDFS-7240 branch into
> trunk. The context can be reviewed in the DISCUSSION thread, and in the
> jiras (See references below).
>
> HDFS-7240 introduces Hadoop Distributed Storage Layer (HDSL), which is
> a distributed, replicated block layer.
> The old HDFS namespace and NN can be connected to this new block layer
> as we have described in HDFS-10419.
> We also introduce a key-value namespace called Ozone built on HDSL.
>

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Owen O'Malley]

I think it would be good to get this in sooner rather than later, but I
have some thoughts.

   1. It is hard to tell what has changed. git rebase -i tells me the
   branch has 722 commits. The rebase failed with a conflict. It would really
   help if you rebased to current trunk.
   2. I think Ozone would be a good Hadoop subproject, but it should be
   outside of HDFS.
   3. CBlock, which is also coming in this merge, would benefit from more
   separation from HDFS.
   4. What are the new transitive dependencies that Ozone, HDSL, and CBlock
   adding to the clients? The servers matter too, but the client dependencies
   have a huge impact on our users.
   5. Have you checked the new dependencies for compatibility with ASL?


On Thu, Mar 1, 2018 at 2:45 PM, Clay B.  wrote:

> Oops, retrying now subscribed to more than solely yarn-dev.
>
> -Clay
>
>
> On Wed, 28 Feb 2018, Clay B. wrote:
>
> +1 (non-binding)
>>
>> I have walked through the code and find it very compelling as a user; I
>> really look forward to seeing the Ozone code mature and it maturing HDFS
>> features together. The points which excite me as an eight year HDFS user
>> are:
>>
>> * Excitement for making the datanode a storage technology container - this
>>  patch clearly brings fresh thought to HDFS keeping it from growing stale
>>
>> * Ability to build upon a shared storage infrastructure for diverse
>>  loads: I do not want to have "stranded" storage capacity or have to
>>  manage competing storage systems on the same disks (and further I want
>>  the metrics datanodes can provide me today, so I do not have to
>>  instrument two systems or evolve their instrumentation separately).
>>
>> * Looking forward to supporting object-sized files!
>>
>> * Moves HDFS in the right direction to test out new block management
>>  techniques for scaling HDFS. I am really excited to see the raft
>>  integration; I hope it opens a new era in Hadoop matching modern systems
>>  design with new consistency and replication options in our ever
>>  distributed ecosystem.
>>
>> -Clay
>>
>> On Mon, 26 Feb 2018, Jitendra Pandey wrote:
>>
>>Dear folks,
>>>   We would like to start a vote to merge HDFS-7240 branch into
>>> trunk. The context can be reviewed in the DISCUSSION thread, and in the
>>> jiras (See references below).
>>>
>>>HDFS-7240 introduces Hadoop Distributed Storage Layer (HDSL), which
>>> is a distributed, replicated block layer.
>>>The old HDFS namespace and NN can be connected to this new block
>>> layer as we have described in HDFS-10419.
>>>We also introduce a key-value namespace called Ozone built on HDSL.
>>>
>>>The code is in a separate module and is turned off by default. In a
>>> secure setup, HDSL and Ozone daemons cannot be started.
>>>
>>>The detailed documentation is available at
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+
>>> Distributed+Storage+Layer+and+Applications
>>>
>>>
>>>I will start with my vote.
>>>+1 (binding)
>>>
>>>
>>>Discussion Thread:
>>>  https://s.apache.org/7240-merge
>>>  https://s.apache.org/4sfU
>>>
>>>Jiras:
>>>   https://issues.apache.org/jira/browse/HDFS-7240
>>>   https://issues.apache.org/jira/browse/HDFS-10419
>>>   https://issues.apache.org/jira/browse/HDFS-13074
>>>   https://issues.apache.org/jira/browse/HDFS-13180
>>>
>>>
>>>Thanks
>>>jitendra
>>>
>>>
>>>
>>>
>>>
>>>DISCUSSION THREAD SUMMARY :
>>>
>>>On 2/13/18, 6:28 PM, "sanjay Radia" 
>>> wrote:
>>>
>>>Sorry the formatting got messed by my email client.  Here
>>> it is again
>>>
>>>
>>>Dear
>>> Hadoop Community Members,
>>>
>>>   We had multiple community discussions, a few meetings
>>> in smaller groups and also jira discussions with respect to this thread. We
>>> express our gratitude for participation and valuable comments.
>>>
>>>The key questions raised were following
>>>1) How the new block storage layer and OzoneFS benefit
>>> HDFS and we were asked to chalk out a roadmap towards the goal of a
>>> scalable namenode working with the new storage layer
>>>2) We were asked to provide a security design
>>>3)There were questions around stability given ozone
>>> brings in a large body of code.
>>>4) Why can?t they be separate projects forever or merged
>>> in when production ready?
>>>
>>>We have responded to all the above questions with
>>> detailed explanations and answers on the jira as well as in the
>>> discussions. We believe that should sufficiently address community?s
>>> concerns.
>>>
>>>Please see the summary below:
>>>
>>>1) The new code base benefits HDFS scaling and a roadmap
>>> has been provided.
>>>
>>>

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Clay B.]

Oops, retrying now subscribed to more than solely yarn-dev.

-Clay

On Wed, 28 Feb 2018, Clay B. wrote:


+1 (non-binding)

I have walked through the code and find it very compelling as a 
user; I really look forward to seeing the Ozone code mature and 
it maturing HDFS features together. The points which excite me 
as an eight year HDFS user are:


* Excitement for making the datanode a storage technology 
container - this
 patch clearly brings fresh thought to HDFS keeping it from 
growing stale


* Ability to build upon a shared storage infrastructure for 
diverse
 loads: I do not want to have "stranded" storage capacity or 
have to
 manage competing storage systems on the same disks (and 
further I want
 the metrics datanodes can provide me today, so I do not have 
to
 instrument two systems or evolve their instrumentation 
separately).


* Looking forward to supporting object-sized files!

* Moves HDFS in the right direction to test out new block 
management
 techniques for scaling HDFS. I am really excited to see the 
raft
 integration; I hope it opens a new era in Hadoop matching 
modern systems
 design with new consistency and replication options in our 
ever

 distributed ecosystem.

-Clay

On Mon, 26 Feb 2018, Jitendra Pandey wrote:


   Dear folks,
  We would like to start a vote to merge HDFS-7240 
branch into trunk. The context can be reviewed in the 
DISCUSSION thread, and in the jiras (See references below).


   HDFS-7240 introduces Hadoop Distributed Storage Layer 
(HDSL), which is a distributed, replicated block layer.
   The old HDFS namespace and NN can be connected to this new 
block layer as we have described in HDFS-10419.
   We also introduce a key-value namespace called Ozone built 
on HDSL.


   The code is in a separate module and is turned off by 
default. In a secure setup, HDSL and Ozone daemons cannot be 
started.


   The detailed documentation is available at

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Distributed+Storage+Layer+and+Applications


   I will start with my vote.
   +1 (binding)


   Discussion Thread:
 https://s.apache.org/7240-merge
 https://s.apache.org/4sfU

   Jiras:
  https://issues.apache.org/jira/browse/HDFS-7240
  https://issues.apache.org/jira/browse/HDFS-10419
  https://issues.apache.org/jira/browse/HDFS-13074
  https://issues.apache.org/jira/browse/HDFS-13180


   Thanks
   jitendra





   DISCUSSION THREAD SUMMARY :

   On 2/13/18, 6:28 PM, "sanjay Radia" 
 wrote:


   Sorry the formatting got messed by my email 
client.  Here it is again



   Dear
Hadoop Community Members,

  We had multiple community discussions, a few 
meetings in smaller groups and also jira discussions with 
respect to this thread. We express our gratitude for 
participation and valuable comments.


   The key questions raised were following
   1) How the new block storage layer and OzoneFS 
benefit HDFS and we were asked to chalk out a roadmap towards 
the goal of a scalable namenode working with the new storage 
layer

   2) We were asked to provide a security design
   3)There were questions around stability given 
ozone brings in a large body of code.
   4) Why can?t they be separate projects forever 
or merged in when production ready?


   We have responded to all the above questions 
with detailed explanations and answers on the jira as well as 
in the discussions. We believe that should sufficiently 
address community?s concerns.


   Please see the summary below:

   1) The new code base benefits HDFS scaling and 
a roadmap has been provided.


   Summary:
 - New block storage layer addresses the 
scalability of the block layer. We have shown how existing NN 
can be connected to the new block layer and its benefits. We 
have shown 2 milestones, 1st milestone is much simpler than 
2nd milestone while giving almost the same scaling benefits. 
Originally we had proposed simply milestone 2 and the 
community felt that removing the FSN/BM lock was was a fair 
amount of work and a simpler solution would be useful
 - We provide a new K-V namespace called Ozone 
FS with FileSystem/FileContext plugins to allow the users to 
use the new system. BTW Hive and Spark work very well on 
KV-namespaces on the cloud. This will facilitate stabilizing 
the new block layer.
 - The new block layer has a new netty based 
protocol engine in the Datanode which, when stabilized, can be 
used by  the old hdfs block layer. See details below on 
sharing of code.



   2) Stability impact on the existing HDFS code 
base and code separation. The new block layer and the OzoneFS 
are in modules that are separate from old HDFS 

Re: [VOTE] Merging branch HDFS-7240 to trunk

[sanjay Radia]

Andrew, thanks for your response.

1) Wrt to NN on top of HDSL. You raised the issue of FSN lock separation . This 
was a key issue we discussed heavily in the past in the context of “Show the 
community a way to connect NN into the the new block layer”. We heard you 
clearly and thought deeply and showed how NN can be put on top of  WITHOUT 
removing the FSN.  We described this in detail  in HDFS-10419 and also  in the 
summary of the DISCUSSION thread:
  Milestone 1 (no removal of FSN) gives almost 2x scalability and does not 
require separation of FSN lock and that milestone 2 which removes the FSN lock 
gives 2x scalability. 

You have conveniently ignored this. Let me reemphasize: Removing the FSN lock 
is not necessary for NN/HDFS to benefit from HDSL and you get almost the same 
scalability benefit. Hence the FSN local issue is moot. 

2) You have also conveniently ignored our arguments that there is benefit in 
keeping HDSL and HDFS together that are in the vote and discussion thread 
summary:
  A) Side by side usage and resulting operational concerns
>>"In the short term and medium term, the new system and HDFS
>> will be used side-by-side by users. ……  
>> During this time, sharing the DN daemon and admin functions
>> between the two systems is operationally important”

   B) Sharing code 
>>"Need to easily share the block layer code between the two systems
>> when used side-by-side. Areas where sharing code is desired over time: 
>>  - Sharing new block layer’s  new netty based protocol engine
>> for old HDFS DNs (a long time sore issue for HDFS block layer). 
>> - Shallow data copy from old system to new system is practical
>> only if within same project and daemon otherwise have to deal
>> with security setting and coordinations across daemons.
>> Shallow copy is useful as customer migrate from old to new.
>> - Shared disk scheduling in the future"



3) You argue for separate project from 2 conflicting arguments: (1) Separate 
then merge later, what’s the hurry.  (2) keep seperate and focus on non-HDFS 
storage use cases. The HDFS community members built HDSL to address HDFS 
scalability; they were  not trying go after object store users or market (ceph 
etc). As explained multiple times OzoneFS is an intermediate step to stabilize 
HDSL but of immediate value for apps such as Hive and Spark. So even if there 
might be value in being separate (your motivation 2)  and go after a new 
storage use cases, the HDFS community members that built HDSL want to focus on 
improving HDFS; you may not agree with that but the engineers that are writing 
the code should be able to drive the direction.  Further look at the Security 
design we posted  - shows a Hadoop/HDFS focus not a focus for some other object 
store market: it fits into the Hadoop security model, especially supporting the 
use case of Jobs and the resulting need to support delegation tokens. 

4) You argue that the  HDSL and OzoneFS modules are separate and therefore one 
should go as a separate project. * Looks like one can’t win here. Damned if you 
do and Damned if you don’t. In the discussion with the Cloudera team one of the 
issues raised was that there a lot of new code and it will destabilized HDFS. 
We explained that  we have kept the code in separate modules so that it will 
not impact current HDFS stability, and that features like HDSL’s  new protocol 
engine will be plugged into the old HDFS block layer only after stabilization. 
You argue for stability and hence separate modules and then use it against to 
push it out as a separate project.

sanjay


> On Feb 28, 2018, at 12:10 AM, Andrew Wang  wrote:
> 
> Resending since the formatting was messed up, let's try plain text this
> time:
> 
> Hi Jitendra and all,
> 
> Thanks for putting this together. I caught up on the discussion on JIRA and
> document at HDFS-10419, and still have the same concerns raised earlier
> about merging the Ozone branch to trunk.
> 
> To recap these questions/concerns at a very high level:
> 
> * Wouldn't Ozone benefit from being a separate project?
> * Why should it be merged now?
> 
> I still believe that both Ozone and Hadoop would benefit from Ozone being a
> separate project, and that there is no pressing reason to merge Ozone/HDSL
> now.
> 
> The primary reason I've heard for merging is that the Ozone is that it's at
> a stage where it's ready for user feedback. Second, that it needs to be
> merged to start on the NN refactoring for HDFS-on-HDSL.
> 
> First, without HDFS-on-HDSL support, users are testing against the Ozone
> object storage interface. Ozone and HDSL themselves are implemented as
> separate masters and new functionality bolted onto the datanode. It also
> doesn't look like HDFS in terms of API or featureset; yes, it speaks
> FileSystem, but so do many out-of-tree storage systems like S3, Ceph,
> Swift, ADLS etc. Ozone/HDSL does not support popular HDFS features like
> erasure coding, encryption, 

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Andrew Wang]

Resending since the formatting was messed up, let's try plain text this
time:

Hi Jitendra and all,

Thanks for putting this together. I caught up on the discussion on JIRA and
document at HDFS-10419, and still have the same concerns raised earlier
about merging the Ozone branch to trunk.

To recap these questions/concerns at a very high level:

* Wouldn't Ozone benefit from being a separate project?
* Why should it be merged now?

I still believe that both Ozone and Hadoop would benefit from Ozone being a
separate project, and that there is no pressing reason to merge Ozone/HDSL
now.

The primary reason I've heard for merging is that the Ozone is that it's at
a stage where it's ready for user feedback. Second, that it needs to be
merged to start on the NN refactoring for HDFS-on-HDSL.

First, without HDFS-on-HDSL support, users are testing against the Ozone
object storage interface. Ozone and HDSL themselves are implemented as
separate masters and new functionality bolted onto the datanode. It also
doesn't look like HDFS in terms of API or featureset; yes, it speaks
FileSystem, but so do many out-of-tree storage systems like S3, Ceph,
Swift, ADLS etc. Ozone/HDSL does not support popular HDFS features like
erasure coding, encryption, high-availability, snapshots, hflush/hsync (and
thus HBase), or APIs like WebHDFS or NFS. This means that Ozone feels like
a new, different system that could reasonably be deployed and tested
separately from HDFS. It's unlikely to replace many of today's HDFS
deployments, and from what I understand, Ozone was not designed to do this.

Second, the NameNode refactoring for HDFS-on-HDSL by itself is a major
undertaking. The discussion on HDFS-10419 is still ongoing so it’s not
clear what the ultimate refactoring will be, but I do know that the earlier
FSN/BM refactoring during 2.x was very painful (introducing new bugs and
making backports difficult) and probably should have been deferred to a new
major release instead. I think this refactoring is important for the
long-term maintainability of the NN and worth pursuing, but as a Hadoop 4.0
item. Merging HDSL is also not a prerequisite for starting this
refactoring. Really, I see the refactoring as the prerequisite for
HDFS-on-HDSL to be possible.

Finally, I earnestly believe that Ozone/HDSL itself would benefit from
being a separate project. Ozone could release faster and iterate more
quickly if it wasn't hampered by Hadoop's release schedule and security and
compatibility requirements. There are also publicity and community
benefits; it's an opportunity to build a community focused on the novel
capabilities and architectural choices of Ozone/HDSL. There are examples of
other projects that were "incubated" on a branch in the Hadoop repo before
being spun off to great success.

In conclusion, I'd like to see Ozone succeeding and thriving as a separate
project. Meanwhile, we can work on the HDFS refactoring required to
separate the FSN and BM and make it pluggable. At that point (likely in the
Hadoop 4 timeframe), we'll be ready to pursue HDFS-on-HDSL integration.

Best,
Andrew

On Tue, Feb 27, 2018 at 11:23 PM, Andrew Wang 
wrote:

>
>
>
>
>
>
>
>
>
> *Hi Jitendra and all,Thanks for putting this together. I caught up on the
> discussion on JIRA and document at HDFS-10419, and still have the same
> concerns raised earlier
> 
> about merging the Ozone branch to trunk.To recap these questions/concerns
> at a very high level:* Wouldn't Ozone benefit from being a separate
> project?* Why should it be merged now?I still believe that both Ozone and
> Hadoop would benefit from Ozone being a separate project, and that there is
> no pressing reason to merge Ozone/HDSL now.The primary reason I've heard
> for merging is that the Ozone is that it's at a stage where it's ready for
> user feedback. Second, that it needs to be merged to start on the NN
> refactoring for HDFS-on-HDSL.First, without HDFS-on-HDSL support, users are
> testing against the Ozone object storage interface. Ozone and HDSL
> themselves are implemented as separate masters and new functionality bolted
> onto the datanode. It also doesn't look like HDFS in terms of API or
> featureset; yes, it speaks FileSystem, but so do many out-of-tree storage
> systems like S3, Ceph, Swift, ADLS etc. Ozone/HDSL does not support popular
> HDFS features like erasure coding, encryption, high-availability,
> snapshots, hflush/hsync (and thus HBase), or APIs like WebHDFS or NFS. This
> means that Ozone feels like a new, different system that could reasonably
> be deployed and tested separately from HDFS. It's unlikely to replace many
> of today's HDFS deployments, and from what I understand, Ozone was not
> designed to do this.Second, the NameNode refactoring for HDFS-on-HDSL by
> itself is a major undertaking. 

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Andrew Wang]

*Hi Jitendra and all,Thanks for putting this together. I caught up on the
discussion on JIRA and document at HDFS-10419, and still have the same
concerns raised earlier

about merging the Ozone branch to trunk.To recap these questions/concerns
at a very high level:* Wouldn't Ozone benefit from being a separate
project?* Why should it be merged now?I still believe that both Ozone and
Hadoop would benefit from Ozone being a separate project, and that there is
no pressing reason to merge Ozone/HDSL now.The primary reason I've heard
for merging is that the Ozone is that it's at a stage where it's ready for
user feedback. Second, that it needs to be merged to start on the NN
refactoring for HDFS-on-HDSL.First, without HDFS-on-HDSL support, users are
testing against the Ozone object storage interface. Ozone and HDSL
themselves are implemented as separate masters and new functionality bolted
onto the datanode. It also doesn't look like HDFS in terms of API or
featureset; yes, it speaks FileSystem, but so do many out-of-tree storage
systems like S3, Ceph, Swift, ADLS etc. Ozone/HDSL does not support popular
HDFS features like erasure coding, encryption, high-availability,
snapshots, hflush/hsync (and thus HBase), or APIs like WebHDFS or NFS. This
means that Ozone feels like a new, different system that could reasonably
be deployed and tested separately from HDFS. It's unlikely to replace many
of today's HDFS deployments, and from what I understand, Ozone was not
designed to do this.Second, the NameNode refactoring for HDFS-on-HDSL by
itself is a major undertaking. The discussion on HDFS-10419 is still
ongoing so it’s not clear what the ultimate refactoring will be, but I do
know that the earlier FSN/BM refactoring during 2.x was very painful
(introducing new bugs and making backports difficult) and probably should
have been deferred to a new major release instead. I think this refactoring
is important for the long-term maintainability of the NN and worth
pursuing, but as a Hadoop 4.0 item. Merging HDSL is also not a prerequisite
for starting this refactoring. Really, I see the refactoring as the
prerequisite for HDFS-on-HDSL to be possible.Finally, I earnestly believe
that Ozone/HDSL itself would benefit from being a separate project. Ozone
could release faster and iterate more quickly if it wasn't hampered by
Hadoop's release schedule and security and compatibility requirements.
There are also publicity and community benefits; it's an opportunity to
build a community focused on the novel capabilities and architectural
choices of Ozone/HDSL. There are examples of other projects that were
"incubated" on a branch in the Hadoop repo before being spun off to great
success.In conclusion, I'd like to see Ozone succeeding and thriving as a
separate project. Meanwhile, we can work on the HDFS refactoring required
to separate the FSN and BM and make it pluggable. At that point (likely in
the Hadoop 4 timeframe), we'll be ready to pursue HDFS-on-HDSL integration.*
Best,
Andrew

On Mon, Feb 26, 2018 at 1:18 PM, Jitendra Pandey 
wrote:

> Dear folks,
>We would like to start a vote to merge HDFS-7240 branch into
> trunk. The context can be reviewed in the DISCUSSION thread, and in the
> jiras (See references below).
>
> HDFS-7240 introduces Hadoop Distributed Storage Layer (HDSL), which is
> a distributed, replicated block layer.
> The old HDFS namespace and NN can be connected to this new block layer
> as we have described in HDFS-10419.
> We also introduce a key-value namespace called Ozone built on HDSL.
>
> The code is in a separate module and is turned off by default. In a
> secure setup, HDSL and Ozone daemons cannot be started.
>
> The detailed documentation is available at
>  https://cwiki.apache.org/confluence/display/HADOOP/
> Hadoop+Distributed+Storage+Layer+and+Applications
>
>
> I will start with my vote.
> +1 (binding)
>
>
> Discussion Thread:
>   https://s.apache.org/7240-merge
>   https://s.apache.org/4sfU
>
> Jiras:
>https://issues.apache.org/jira/browse/HDFS-7240
>https://issues.apache.org/jira/browse/HDFS-10419
>https://issues.apache.org/jira/browse/HDFS-13074
>https://issues.apache.org/jira/browse/HDFS-13180
>
>
> Thanks
> jitendra
>
>
>
>
>
> DISCUSSION THREAD SUMMARY :
>
> On 2/13/18, 6:28 PM, "sanjay Radia" 
> wrote:
>
> Sorry the formatting got messed by my email client.  Here
> it is again
>
>
> Dear
>  Hadoop Community Members,
>
>We had multiple community discussions, a few meetings
> in smaller groups and also jira discussions with respect to 

[VOTE] Merging branch HDFS-7240 to trunk

[Jitendra Pandey]

Dear folks,
   We would like to start a vote to merge HDFS-7240 branch into trunk. 
The context can be reviewed in the DISCUSSION thread, and in the jiras (See 
references below).
  
HDFS-7240 introduces Hadoop Distributed Storage Layer (HDSL), which is a 
distributed, replicated block layer.
The old HDFS namespace and NN can be connected to this new block layer as 
we have described in HDFS-10419.
We also introduce a key-value namespace called Ozone built on HDSL.
  
The code is in a separate module and is turned off by default. In a secure 
setup, HDSL and Ozone daemons cannot be started.

The detailed documentation is available at 
 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Distributed+Storage+Layer+and+Applications


I will start with my vote.
+1 (binding)


Discussion Thread:
  https://s.apache.org/7240-merge
  https://s.apache.org/4sfU

Jiras:
   https://issues.apache.org/jira/browse/HDFS-7240
   https://issues.apache.org/jira/browse/HDFS-10419
   https://issues.apache.org/jira/browse/HDFS-13074
   https://issues.apache.org/jira/browse/HDFS-13180

   
Thanks
jitendra





DISCUSSION THREAD SUMMARY :

On 2/13/18, 6:28 PM, "sanjay Radia"  wrote:

Sorry the formatting got messed by my email client.  Here it is 
again


Dear
 Hadoop Community Members,

   We had multiple community discussions, a few meetings in 
smaller groups and also jira discussions with respect to this thread. We 
express our gratitude for participation and valuable comments. 

The key questions raised were following
1) How the new block storage layer and OzoneFS benefit HDFS and 
we were asked to chalk out a roadmap towards the goal of a scalable namenode 
working with the new storage layer
2) We were asked to provide a security design
3)There were questions around stability given ozone brings in a 
large body of code.
4) Why can’t they be separate projects forever or merged in 
when production ready?

We have responded to all the above questions with detailed 
explanations and answers on the jira as well as in the discussions. We believe 
that should sufficiently address community’s concerns. 

Please see the summary below:

1) The new code base benefits HDFS scaling and a roadmap has 
been provided. 

Summary:
  - New block storage layer addresses the scalability of the 
block layer. We have shown how existing NN can be connected to the new block 
layer and its benefits. We have shown 2 milestones, 1st milestone is much 
simpler than 2nd milestone while giving almost the same scaling benefits. 
Originally we had proposed simply milestone 2 and the community felt that 
removing the FSN/BM lock was was a fair amount of work and a simpler solution 
would be useful
  - We provide a new K-V namespace called Ozone FS with 
FileSystem/FileContext plugins to allow the users to use the new system. BTW 
Hive and Spark work very well on KV-namespaces on the cloud. This will 
facilitate stabilizing the new block layer. 
  - The new block layer has a new netty based protocol engine 
in the Datanode which, when stabilized, can be used by  the old hdfs block 
layer. See details below on sharing of code.


2) Stability impact on the existing HDFS code base and code 
separation. The new block layer and the OzoneFS are in modules that are 
separate from old HDFS code - currently there are no calls from HDFS into Ozone 
except for DN starting the new block  layer module if configured to do so. It 
does not add instability (the instability argument has been raised many times). 
Over time as we share code, we will ensure that the old HDFS continues to 
remains stable. (for example we plan to stabilize the new netty based protocol 
engine in the new block layer before sharing it with HDFS’s old block layer)


3) In the short term and medium term, the new system and HDFS  
will be used side-by-side by users. Side by-side usage in the short term for 
testing and side-by-side in the medium term for actual production use till the 
new system has feature parity with old HDFS. During this time, sharing the DN 
daemon and admin functions between the two systems is operationally important: