Re: [VOTE] Merging branch HDFS-7240 to trunk

[sanjay Radia]

> On Mar 5, 2018, at 4:08 PM, Andrew Wang  wrote:
> 
> - NN on top HDSL where the NN uses the new block layer (Both Daryn and Owen 
> acknowledge the benefit of the new block layer).  We have two choices here
>  ** a) Evolve NN so that it can interact with both old and new block layer,
>  **  b) Fork and create new NN that works only with new block layer, the old 
> NN will continue to work with old block layer.
> There are trade-offs but clearly the 2nd option has least impact on the old 
> HDFS code.
> 
> Are you proposing that we pursue the 2nd option to integrate HDSL with HDFS?


Originally I would have prefered (a); but Owen made a strong case for (b) in my 
discussions with his last week.
Overall we need a broader discussion around the next steps for NN evolution and 
how to chart the course; I am not locked into any particular path or how we 
would do it. 
Let me make a more detailed response in HDFS-10419.

sanjay

Re: [VOTE] Merging branch HDFS-7240 to trunk

[sanjay Radia]

Joep,  You raise a number of points:

(1) Ozone vs and object stores. “Some users would choose Ozone as that layer, 
some might use S3, others GCS, or Azure, or something else”.
(2) How HDSL/Ozone fits into Hadoop and whether it is necessary.
(3) You raise the release issue which we will respond in a separate email.

Let me respond to 1 & 2:
***Wrt to (1) Ozone vs other object stores***
Neither HDFS or Ozone has any real role in cloud except for temp data. The cost 
of local disk or EBS is so high that long term data storage on HDFS or even 
Ozone is prohibitive.
So why the hell create the KV namespace? We need to stabilize the HDSL where 
data is stored.  - We are targeting Hive and SPark apps to stabilize HDSL using 
real Hadoop apps over OzoneFS.
But HDSL/Ozone is not feature compatible with HDFS so how will users even use 
it for real to stability. Users can run HDFS and Ozone side by side in same 
cluster and have two namespace (just like in Federation) and run apps on both: 
run some hive and spark apps on Ozone and others that need full HDFS feature 
(encryption) on HDFS. As it becomes stable they can start using HDSL/Ozone for 
production use for a portion of their data.



***Wrt to (2) HDSL/Ozone fitting into Hadoop and why the same repository***
Ozone KV is a temporary step. Real goal is to put NN on top of HDSL, We have 
shown how to do that in the roadmap that Konstantine and Chris D asked. 
Milestone 1 is feasible and doesn't require removal of FSN lock. We have also 
shown several cases of sharing other code in future (protocol engine). This 
co-development will be easier if in the same repo. Over time HDSL + ported NN  
will create a new HDFS and become feature compatible - some of the feature will 
come for free because they are in NN and will port over to the new NN, Some are 
in block layer (erasure code) and will have to be added to HDSL.

--- You compare with Yarn, HDFS and Common. HDFS and Yarn are independent but 
both depend on Hadoop common (e.g. HBase runs on HDFS without Yarn).   HDSL and 
Ozone will depend on Hadoop common, Indeed the new protocol engine of HDSL 
might move to Hadoop common or HDFS. We have made sure that there are no 
dependencies of HDFS on HDSL or currently.


***The Repo issue and conclusion***
HDFS community will need to work together as we evolve old HDFS to use HDSL, 
new protocol engine and Raft. and together evolve to a newer more powerful set 
of sub components. It is important that they are in same repo and that we can 
share code through both private interface. We are not trying to build a 
competing Object store but to improve HDFS and fixing scalability fundamentally 
is hard and we are asking for an environment for that to happen easily over the 
next year while heeding to the stability concerns of HDFS developers (eg we  
remove compile time dependency, maven profile). This work is not being done by 
members of foreign project trying to insert code in Hadoop, but by Hadoop/HDFS 
developers with given track record s and are active participation in Hadoop and 
HDFS. Our jobs depend on HDFS/Hadoop stability - destabilizing is the last 
thing we want to do; we have responded every constructive feedback 


sanjay


> On Mar 6, 2018, at 6:50 PM, J. Rottinghuis  wrote:
> 
> Sorry for jumping in late into the fray of this discussion.
> 
> It seems Ozone is a large feature. I appreciate the development effort and
> the desire to get this into the hands of users.
> I understand the need to iterate quickly and to reduce overhead for
> development.
> I also agree that Hadoop can benefit from a quicker release cycle. For our
> part, this is a challenge as we have a large installation with multiple
> clusters and thousands of users. It is a constant balance between jumping
> to the newest release and the cost of this integration and test at our
> scale, especially when things aren't backwards compatible. We try to be
> good citizens and upstream our changes and contribute back.
> 
> The point was made that splitting the projects such as common and Yarn
> didn't work and had to be reverted. That was painful and a lot of work for
> those involved for sure. This project may be slightly different in that
> hadoop-common, Yarn and HDFS made for one consistent whole. One couldn't
> run a project without the other.
> 
> Having a separate block management layer with possibly multiple block
> implementation as pluggable under the covers would be a good future
> development for HDFS. Some users would choose Ozone as that layer, some
> might use S3, others GCS, or Azure, or something else.
> If the argument is made that nobody will be able to run Hadoop as a
> consistent stack without Ozone, then that would be a strong case to keep
> things in the same repo.
> 
> Obviously when people do want to use Ozone, then having it in the same repo
> is easier. The flipside is that, separate top-level project in the same
> repo or not, it adds to the Hadoop 

Re: [VOTE] Merging branch HDFS-7240 to trunk

[sanjay Radia]

 hdfs but ignore it due to anticipation of its
>> departure/demise.  I’m not implying that’s currently happening, it’s just
>> what I don’t want to see.
>> 
>> 
>> We as a community and our customers need an evolution, not a revolution,
>> and definitively not a civil war.  Hdfs has too much legacy code rot that
>> is hard to change.  Too many poorly implemented features.   Perhaps I’m
>> overly optimistic that freshly redesigned code can counterbalance
>> performance degradations in the NN.  I’m also reluctant, but realize it is
>> being driven by some hdfs veterans that know/understand historical hdfs
>> design strengths and flaws.
>> 
>> 
>> If the initially cited issues are addressed, I’m +0.5 for the concept of
>> bringing in ozone if it's not going to be a proverbial bull in the china
>> shop.
>> 
>> 
>> Daryn
>> 
>> On Mon, Feb 26, 2018 at 3:18 PM, Jitendra Pandey <jiten...@hortonworks.com
>>> 
>> wrote:
>> 
>>>Dear folks,
>>>   We would like to start a vote to merge HDFS-7240 branch into
>>> trunk. The context can be reviewed in the DISCUSSION thread, and in the
>>> jiras (See references below).
>>> 
>>>HDFS-7240 introduces Hadoop Distributed Storage Layer (HDSL), which
>> is
>>> a distributed, replicated block layer.
>>>The old HDFS namespace and NN can be connected to this new block
>> layer
>>> as we have described in HDFS-10419.
>>>We also introduce a key-value namespace called Ozone built on HDSL.
>>> 
>>>The code is in a separate module and is turned off by default. In a
>>> secure setup, HDSL and Ozone daemons cannot be started.
>>> 
>>>The detailed documentation is available at
>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> Hadoop+Distributed+Storage+Layer+and+Applications
>>> 
>>> 
>>>I will start with my vote.
>>>+1 (binding)
>>> 
>>> 
>>>Discussion Thread:
>>>  https://s.apache.org/7240-merge
>>>  https://s.apache.org/4sfU
>>> 
>>>Jiras:
>>>   https://issues.apache.org/jira/browse/HDFS-7240
>>>   https://issues.apache.org/jira/browse/HDFS-10419
>>>   https://issues.apache.org/jira/browse/HDFS-13074
>>>   https://issues.apache.org/jira/browse/HDFS-13180
>>> 
>>> 
>>>Thanks
>>>jitendra
>>> 
>>> 
>>> 
>>> 
>>> 
>>>DISCUSSION THREAD SUMMARY :
>>> 
>>>On 2/13/18, 6:28 PM, "sanjay Radia" <sanjayo...@gmail.com>
>>> wrote:
>>> 
>>>Sorry the formatting got messed by my email client.  Here
>>> it is again
>>> 
>>> 
>>>Dear
>>> Hadoop Community Members,
>>> 
>>>   We had multiple community discussions, a few meetings
>>> in smaller groups and also jira discussions with respect to this thread.
>> We
>>> express our gratitude for participation and valuable comments.
>>> 
>>>The key questions raised were following
>>>1) How the new block storage layer and OzoneFS benefit
>>> HDFS and we were asked to chalk out a roadmap towards the goal of a
>>> scalable namenode working with the new storage layer
>>>2) We were asked to provide a security design
>>>3)There were questions around stability given ozone
>> brings
>>> in a large body of code.
>>>4) Why can’t they be separate projects forever or merged
>>> in when production ready?
>>> 
>>>We have responded to all the above questions with
>> detailed
>>> explanations and answers on the jira as well as in the discussions. We
>>> believe that should sufficiently address community’s concerns.
>>> 
>>>Please see the summary below:
>>> 
>>>1) The new code base benefits HDFS scaling and a roadmap
>>> has been provided.
>>> 
>>>Summary:
>>>  - New block storage layer addresses the scalability of
>>> the block layer. We have shown how existing NN can be connected to the
>> new
>>> block layer and its benefits. We have shown 2 milestones, 1st milestone
>> is
>>

Re: [VOTE] Merging branch HDFS-7240 to trunk

[sanjay Radia]

equisite for HDFS-on-HDSL to be possible.Finally, I earnestly believe
>> that Ozone/HDSL itself would benefit from being a separate project. Ozone
>> could release faster and iterate more quickly if it wasn't hampered by
>> Hadoop's release schedule and security and compatibility requirements.
>> There are also publicity and community benefits; it's an opportunity to
>> build a community focused on the novel capabilities and architectural
>> choices of Ozone/HDSL. There are examples of other projects that were
>> "incubated" on a branch in the Hadoop repo before being spun off to great
>> success.In conclusion, I'd like to see Ozone succeeding and thriving as a
>> separate project. Meanwhile, we can work on the HDFS refactoring required
>> to separate the FSN and BM and make it pluggable. At that point (likely in
>> the Hadoop 4 timeframe), we'll be ready to pursue HDFS-on-HDSL integration.*
>> Best,
>> Andrew
>> 
>> On Mon, Feb 26, 2018 at 1:18 PM, Jitendra Pandey <jiten...@hortonworks.com
>>> wrote:
>> 
>>>Dear folks,
>>>   We would like to start a vote to merge HDFS-7240 branch into
>>> trunk. The context can be reviewed in the DISCUSSION thread, and in the
>>> jiras (See references below).
>>> 
>>>HDFS-7240 introduces Hadoop Distributed Storage Layer (HDSL), which
>>> is a distributed, replicated block layer.
>>>The old HDFS namespace and NN can be connected to this new block
>>> layer as we have described in HDFS-10419.
>>>We also introduce a key-value namespace called Ozone built on HDSL.
>>> 
>>>The code is in a separate module and is turned off by default. In a
>>> secure setup, HDSL and Ozone daemons cannot be started.
>>> 
>>>The detailed documentation is available at
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+
>>> Distributed+Storage+Layer+and+Applications
>>> 
>>> 
>>>I will start with my vote.
>>>+1 (binding)
>>> 
>>> 
>>>Discussion Thread:
>>>  https://s.apache.org/7240-merge
>>>  https://s.apache.org/4sfU
>>> 
>>>Jiras:
>>>   https://issues.apache.org/jira/browse/HDFS-7240
>>>   https://issues.apache.org/jira/browse/HDFS-10419
>>>   https://issues.apache.org/jira/browse/HDFS-13074
>>>   https://issues.apache.org/jira/browse/HDFS-13180
>>> 
>>> 
>>>Thanks
>>>jitendra
>>> 
>>> 
>>> 
>>> 
>>> 
>>>DISCUSSION THREAD SUMMARY :
>>> 
>>>On 2/13/18, 6:28 PM, "sanjay Radia" <sanjayo...@gmail.com>
>>> wrote:
>>> 
>>>Sorry the formatting got messed by my email client.  Here
>>> it is again
>>> 
>>> 
>>>Dear
>>> Hadoop Community Members,
>>> 
>>>   We had multiple community discussions, a few meetings
>>> in smaller groups and also jira discussions with respect to this thread. We
>>> express our gratitude for participation and valuable comments.
>>> 
>>>The key questions raised were following
>>>1) How the new block storage layer and OzoneFS benefit
>>> HDFS and we were asked to chalk out a roadmap towards the goal of a
>>> scalable namenode working with the new storage layer
>>>2) We were asked to provide a security design
>>>3)There were questions around stability given ozone
>>> brings in a large body of code.
>>>4) Why can’t they be separate projects forever or merged
>>> in when production ready?
>>> 
>>>We have responded to all the above questions with
>>> detailed explanations and answers on the jira as well as in the
>>> discussions. We believe that should sufficiently address community’s
>>> concerns.
>>> 
>>>Please see the summary below:
>>> 
>>>1) The new code base benefits HDFS scaling and a roadmap
>>> has been provided.
>>> 
>>>Summary:
>>>  - New block storage layer addresses the scalability of
>>> the block layer. We have shown how existing NN can be connected to the new
>>> block layer and its benefits. We have shown 2 mileston

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[sanjay Radia]

Konstantine


Thanks for your feedback and comments over the last few months.  Have we 
addressed all your  issues and concerns?

sanjay


> On Feb 13, 2018, at 6:28 PM, sanjay Radia <sanjayo...@gmail.com> wrote:
> 
> Sorry the formatting got messed by my email client.  Here it is again
> 
> 
> Dear
> Hadoop Community Members,
> 
>  We had multiple community discussions, a few meetings in smaller groups and 
> also jira discussions with respect to this thread. We express our gratitude 
> for participation and valuable comments. 
> 
> The key questions raised were following
> 1) How the new block storage layer and OzoneFS benefit HDFS and we were asked 
> to chalk out a roadmap towards the goal of a scalable namenode working with 
> the new storage layer
> 2) We were asked to provide a security design
> 3)There were questions around stability given ozone brings in a large body of 
> code.
> 4) Why can’t they be separate projects forever or merged in when production 
> ready?
> 
> We have responded to all the above questions with detailed explanations and 
> answers on the jira as well as in the discussions. We believe that should 
> sufficiently address community’s concerns. 
> 
> Please see the summary below:
> 
> 1) The new code base benefits HDFS scaling and a roadmap has been provided. 
> 
> Summary:
> - New block storage layer addresses the scalability of the block layer. We 
> have shown how existing NN can be connected to the new block layer and its 
> benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd 
> milestone while giving almost the same scaling benefits. Originally we had 
> proposed simply milestone 2 and the community felt that removing the FSN/BM 
> lock was was a fair amount of work and a simpler solution would be useful
> - We provide a new K-V namespace called Ozone FS with FileSystem/FileContext 
> plugins to allow the users to use the new system. BTW Hive and Spark work 
> very well on KV-namespaces on the cloud. This will facilitate stabilizing the 
> new block layer. 
> - The new block layer has a new netty based protocol engine in the Datanode 
> which, when stabilized, can be used by  the old hdfs block layer. See details 
> below on sharing of code.
> 
> 
> 2) Stability impact on the existing HDFS code base and code separation. The 
> new block layer and the OzoneFS are in modules that are separate from old 
> HDFS code - currently there are no calls from HDFS into Ozone except for DN 
> starting the new block  layer module if configured to do so. It does not add 
> instability (the instability argument has been raised many times). Over time 
> as we share code, we will ensure that the old HDFS continues to remains 
> stable. (for example we plan to stabilize the new netty based protocol engine 
> in the new block layer before sharing it with HDFS’s old block layer)
> 
> 
> 3) In the short term and medium term, the new system and HDFS  will be used 
> side-by-side by users. Side by-side usage in the short term for testing and 
> side-by-side in the medium term for actual production use till the new system 
> has feature parity with old HDFS. During this time, sharing the DN daemon and 
> admin functions between the two systems is operationally important:  
> - Sharing DN daemon to avoid additional operational daemon lifecycle 
> management
> - Common decommissioning of the daemon and DN: One place to decommission for 
> a node and its storage.
> - Replacing failed disks and internal balancing capacity across disks - this 
> needs to be done for both the current HDFS blocks and the new block-layer 
> blocks.
> - Balancer: we would like use the same balancer and provide a common way to 
> balance and common management of the bandwidth used for balancing
> - Security configuration setup - reuse existing set up for DNs rather then a 
> new one for an independent cluster.
> 
> 
> 4) Need to easily share the block layer code between the two systems when 
> used side-by-side. Areas where sharing code is desired over time: 
> - Sharing new block layer’s  new netty based protocol engine for old HDFS DNs 
> (a long time sore issue for HDFS block layer). 
> - Shallow data copy from old system to new system is practical only if within 
> same project and daemon otherwise have to deal with security setting and 
> coordinations across daemons. Shallow copy is useful as customer migrate from 
> old to new.
> - Shared disk scheduling in the future and in the short term have a single 
> round robin rather than independent round robins.
> While sharing code across projects is technically possible (anything is 
> possible in software),  it is significantly harder typically requiring  
> cleaner public apis e

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[sanjay Radia]

Sorry the formatting got messed by my email client.  Here it is again


Dear
 Hadoop Community Members,

   We had multiple community discussions, a few meetings in smaller groups and 
also jira discussions with respect to this thread. We express our gratitude for 
participation and valuable comments. 

The key questions raised were following
1) How the new block storage layer and OzoneFS benefit HDFS and we were asked 
to chalk out a roadmap towards the goal of a scalable namenode working with the 
new storage layer
2) We were asked to provide a security design
3)There were questions around stability given ozone brings in a large body of 
code.
4) Why can’t they be separate projects forever or merged in when production 
ready?

We have responded to all the above questions with detailed explanations and 
answers on the jira as well as in the discussions. We believe that should 
sufficiently address community’s concerns. 

Please see the summary below:

1) The new code base benefits HDFS scaling and a roadmap has been provided. 

Summary:
  - New block storage layer addresses the scalability of the block layer. We 
have shown how existing NN can be connected to the new block layer and its 
benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd 
milestone while giving almost the same scaling benefits. Originally we had 
proposed simply milestone 2 and the community felt that removing the FSN/BM 
lock was was a fair amount of work and a simpler solution would be useful
  - We provide a new K-V namespace called Ozone FS with FileSystem/FileContext 
plugins to allow the users to use the new system. BTW Hive and Spark work very 
well on KV-namespaces on the cloud. This will facilitate stabilizing the new 
block layer. 
  - The new block layer has a new netty based protocol engine in the Datanode 
which, when stabilized, can be used by  the old hdfs block layer. See details 
below on sharing of code.


2) Stability impact on the existing HDFS code base and code separation. The new 
block layer and the OzoneFS are in modules that are separate from old HDFS code 
- currently there are no calls from HDFS into Ozone except for DN starting the 
new block  layer module if configured to do so. It does not add instability 
(the instability argument has been raised many times). Over time as we share 
code, we will ensure that the old HDFS continues to remains stable. (for 
example we plan to stabilize the new netty based protocol engine in the new 
block layer before sharing it with HDFS’s old block layer)


3) In the short term and medium term, the new system and HDFS  will be used 
side-by-side by users. Side by-side usage in the short term for testing and 
side-by-side in the medium term for actual production use till the new system 
has feature parity with old HDFS. During this time, sharing the DN daemon and 
admin functions between the two systems is operationally important:  
  - Sharing DN daemon to avoid additional operational daemon lifecycle 
management
  - Common decommissioning of the daemon and DN: One place to decommission for 
a node and its storage.
  - Replacing failed disks and internal balancing capacity across disks - this 
needs to be done for both the current HDFS blocks and the new block-layer 
blocks.
  - Balancer: we would like use the same balancer and provide a common way to 
balance and common management of the bandwidth used for balancing
  - Security configuration setup - reuse existing set up for DNs rather then a 
new one for an independent cluster.


4) Need to easily share the block layer code between the two systems when used 
side-by-side. Areas where sharing code is desired over time: 
  - Sharing new block layer’s  new netty based protocol engine for old HDFS DNs 
(a long time sore issue for HDFS block layer). 
  - Shallow data copy from old system to new system is practical only if within 
same project and daemon otherwise have to deal with security setting and 
coordinations across daemons. Shallow copy is useful as customer migrate from 
old to new.
  - Shared disk scheduling in the future and in the short term have a single 
round robin rather than independent round robins.
While sharing code across projects is technically possible (anything is 
possible in software),  it is significantly harder typically requiring  cleaner 
public apis etc. Sharing within a project though internal APIs is often simpler 
(such as the protocol engine that we want to share).


5) Security design, including a threat model and and the solution has been 
posted.
6) Temporary Separation and merge later: Several of the comments in the jira 
have argued that we temporarily separate the two code bases for now and then 
later merge them when the new code is stable:

  - If there is agreement to merge later, why bother separating now - there 
needs to be to be good reasons to separate now.  We have addressed the 
stability and separation of the new code from existing above.
  - Merge 

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[sanjay Radia]

Dear Hadoop Community Members,

We had multiple community discussions, a few meetings in smaller groups and 
also jira discussions with respect to this thread. We express our gratitude for 
participation and valuable comments. The key questions raised were following 
How the new block storage layer and OzoneFS benefit HDFS and we were asked to 
chalk out a roadmap towards the goal of a scalable namenode working with the 
new storage layer
We were asked to provide a security design
There were questions around stability given ozone brings in a large body of 
code.
Why can’t they be separate projects forever or merged in when production ready?

We have responded to all the above questions with detailed explanations and 
answers on the jira as well as in the discussions. We believe that should 
sufficiently address community’s concerns. Please see the summary below:

The new code base benefits to HDFS scaling and a roadmap has been provided. 
Summary:
New block storage layer addresses the scalability of the block layer. We have 
shown how existing NN can be connected to the new block layer and its benefits. 
We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone 
while giving almost the same scaling benefits.  Originally we had proposed 
simply milestone 2 and the community felt that removing the FSN/BM lock was was 
a fair amount of work and a simpler solution would be useful.
We provide a new K-V namespace called Ozone FS with FileSystem/FileContext 
plugins to allow the users to use the new system. BTW Hive and Spark work very 
well on KV-namespaces on the cloud. This will facilitate stabilizing the new 
block layer. 
The new block layer has a new netty based protocol engine in the Datanode 
which, when stabilized, can be used by the old hdfs block layer. See details 
below on sharing of code.
Stability impact on the existing HDFS code base and code separation. The new 
block layer and the OzoneFS are in modules that are separate from old HDFS code 
- currently there are no calls from HDFS into Ozone except for DN starting the 
new block  layer module if configured to do so. It does not add instability 
(the instability argument has been raised many times). Over time as we share 
code, we will ensure that the old HDFS continues to remains stable. (for 
example we plan to stabilize the new netty based protocol engine in the new 
block layer before sharing it with HDFS’s old block layer)
In the short term and medium term, the new system and HDFS  will be used 
side-by-side by users. Side by-side usage in the short term for testing and 
side-by-side in the medium term for actual production use till the new system 
has feature parity with old HDFS. During this time, sharing the DN daemon and 
admin functions between the two systems is operationally important:  
Sharing DN daemon to avoid additional operational daemon lifecycle management
Common decommissioning of the daemon and DN: One place to decommission for a 
node and its storage.
Replacing failed disks and internal balancing capacity across disks - this 
needs to be done for both the current HDFS blocks and the new block-layer 
blocks.
Balancer: we would like use the same balancer and provide a common way to 
balance and common management of the bandwidth used for balancing
Security configuration setup - reuse existing set up for DNs rather then a new 
one for an independent cluster.
Need to easily share the block layer code between the two systems when used 
side-by-side. Areas where sharing code is desired over time: 
Sharing new block layer’s  new netty based protocol engine for old HDFS DNs (a 
long time sore issue for HDFS block layer). 
Shallow data copy from old system to new system is practical only if within 
same project and daemon otherwise have to deal with security setting and 
coordinations across daemons. Shallow copy is useful as customer migrate from 
old to new.
Shared disk scheduling in the future and in the short term have a single round 
robin rather than independent round robins.
While sharing code across projects is technically possible (anything is 
possible in software),  it is significantly harder typically requiring  cleaner 
public apis etc. Sharing within a project though internal APIs is often simpler 
(such as the protocol engine that we want to share).
Security design, including a threat model and and the solution has been posted.
Temporary Separation and merge later: Several of the comments in the jira have 
argued that we temporarily separate the two code bases for now and then later 
merge them when the new code is stable:
If there is agreement to merge later, why bother separating now - there needs 
to be to be good reasons to separate now.  We have addressed the stability and 
separation of the new code from existing above.
Merge the new code back into HDFS later will be harder. 
The code and goals will diverge further. 
We will be taking on extra work to split and then take extra work to 

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[sanjay Radia]

Konstantine, 
 Thanks for your comments, questions and feedback. I have attached a document 
to the HDFS-7240 jira 
 that explains a design for scaling HDFS and how Ozone paves the way towards 
the full solution.


https://issues.apache.org/jira/secure/attachment/12895963/HDFS%20Scalability%20and%20Ozone.pdf


sanjay




> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko  wrote:
> 
> Hey guys,
> 
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
> 
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
> 
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-8.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
>  a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
>  b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
> 
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
> 
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
> 
> Thanks,
> --Konstantin
> 
> 
> 
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei  wrote:
> 
>> Hello everyone,
>> 
>> 
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>> 
>> 
>> 
>> The merge payload includes the following:
>> 
>>  1.  All services, management scripts
>>  2.  Object store APIs, exposed via both REST and RPC
>>  3.  Master service UIs, command line interfaces
>>  4.  Pluggable pipeline Integration
>>  5.  Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>>  6.  Corona - a load generator for Ozone.
>>  7.  Essential documentation added to Hadoop site.
>>  8.  Version specific Ozone Documentation, accessible via service UI.
>>  9.  Docker support for ozone, which enables faster development cycles.
>> 
>> 
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confl
>> uence/display/HADOOP/Dev+cluster+with+docker.
>> 
>> 
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>> 
>> 
>> Please share your thoughts, thanks!
>> 
>> 
>> -- Weiwei Yang
>> 
> 
> 
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei  wrote:
> 
>> Hello everyone,
>> 
>> 
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>> 
>> 
>> 
>> The merge payload includes the following:
>> 
>>  1.  All services, management scripts
>>  2.  Object store APIs, exposed via both REST and RPC
>>  3.  Master service UIs, command line interfaces
>>  4.  Pluggable pipeline Integration

Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)

[sanjay Radia]

Erik
  Great stuff. 
BTW did you build on top of the “simulated data nodes” in HDFS which has a way 
to storing only the length of data (but not real data)? That work allowed 
supplementing  with a matching editsLog for the NN. Your approach of using a 
real image has the advantage of being able to replay traces from audit logs.
(Ref 
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java)

thanks

sanjay
> On Jul 20, 2017, at 10:42 AM, Erik Krogen  
> wrote:
> 
> forking off of the 2.7.4 release thread to answer this question about
> Dynamometer
> 
> Dynamometer is a tool developed at LinkedIn for scale testing HDFS,
> specifically the NameNode. We have been using it for some time now and have
> recently been making some enhancements to ease of use and reproducibility.
> We hope to post a blog post sometime in the not-too-distant future, and
> also to open source it. I can provide some details here given that we have
> been leveraging it as part of our 2.7.4 release / upgrade process (in
> addition to previous upgrades).
> 
> The basic idea is to get full-scale black-box testing of the HDFS NN while
> using significantly less (~10%) hardware than a real cluster of that size
> would require. We use real NN images from our at-scale clusters paired with
> some logic to fake out DNs into thinking they are storing data when they
> are not, allowing us to stuff more DNs onto each machine. Since we use a
> real image, we can replay real traces (collected from audit logs) to
> compare actual production performance vs. performance on this simulated
> cluster (with additional tuning, different version, etc.). We leverage YARN
> to manage setting up this cluster and to replay the traces.
> 
> Happy to answer questions.
> 
> Erik
> 
> On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko 
> wrote:
> 
>> Hi Tianyi,
>> 
>> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working
>> on this project right now, I'll let him elaborate.
>> Erik, you should probably respond on Apache dev list, as I think it could
>> be interesting for other people as well, asince we planned to open source
>> it. You can fork the "About 2.7.4 Release" thread with a new subject and
>> give some details about Dynamometer there.
>> 
>> Thanks,
>> --Konstantin
>> 
>> On Wed, Jul 19, 2017 at 1:40 AM, 何天一  wrote:
>> 
>>> Hi, Shavachko.
>>> 
>>> You mentioned an internal tool called Dynamometer to test NameNode
>>> performance earlier in the 2.7.4 release thread.
>>> I wonder if you could share some ideas behind the tool. Or is there a
>>> plan to bring Dynamometer to open source community?
>>> 
>>> Thanks.
>>> 
>>> BR,
>>> Tianyi
>>> 
>>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko 
>>> wrote:
>>> 
 Hi everybody.
 
 We have been doing some internal testing of Hadoop 2.7.4. The testing is
 going well.
 Did not find any major issues on our workloads.
 Used an internal tool called Dynamometer to check NameNode performance on
 real cluster traces. Good.
 Overall test cluster performance looks good.
 Some more testing is still going on.
 
 I plan to build an RC next week. If there are no objection.
 
 Thanks,
 --Konst
 
 On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <
 shv.had...@gmail.com>
 wrote:
 
> Hey guys.
> 
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
> 
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I
 filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
> 
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
> 
> There is a plan to enable BigTop for 2.7.4 testing.
> 
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
> 
> Regards,
> --Konstantin
> 
> 
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka 
> wrote:
> 
>> Sure.
>> If you want to edit the wiki, please tell me your ASF confluence
 account.
>> 
>> -Akira
>> 
>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>> 
>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These
 will
>>> solve RM HA unstability issues.
>>> 

Re: Looking to a Hadoop 3 release

[sanjay Radia]

 On Mar 5, 2015, at 3:21 PM, Siddharth Seth ss...@apache.org wrote:
 
 2) Simplification of configs - potentially separating client side configs
 and those used by daemons. This is another source of perpetual confusion
 for users.
+ 1 on this.

sanjay

Re: Looking to a Hadoop 3 release

[sanjay Radia]

 On Mar 3, 2015, at 9:36 AM, Karthik Kambatla ka...@cloudera.com wrote:
 
 If we preserve API compat and try to preserve wire compat, I don't see the
 harm in bumping the major release.

If we preserve compatibility, then there is no need to bump major number.
 It allows us to include several
 fixes/features in trunk in a release. If we are not actively thinking of a
 way to release items in trunk, why even have it?

What are the fixes and features in trunk that you would like to see get out 
quickly?
Can these be back ported easily to branch 2?

sanjay

Re: Looking to a Hadoop 3 release

[sanjay Radia]

Andrew 
  Thanks for bringing up the issue of moving to Java8. Java8 is important
However, I am not seeing a strong motivation for changing the major number.
We can go to Java8 in  the 2.series. 
The classpath issue for Hadoop-11656 is too minor to force a major number 
change (no pun intended).

Lets separate the issue of Java8 and Hadoop 3.0

sanjay


 On Mar 2, 2015, at 3:19 PM, Andrew Wang andrew.w...@cloudera.com wrote:
 
 Hi devs,
 
 It's been a year and a half since 2.x went GA, and I think we're about due
 for a 3.x release.
 Notably, there are two incompatible changes I'd like to call out, that will
 have a tremendous positive impact for our users.
 
 First, classpath isolation being done at HADOOP-11656, which has been a
 long-standing request from many downstreams and Hadoop users.
 
 Second, bumping the source and target JDK version to JDK8 (related to
 HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two
 months from now). In the past, we've had issues with our dependencies
 discontinuing support for old JDKs, so this will future-proof us.
 
 Between the two, we'll also have quite an opportunity to clean up and
 upgrade our dependencies, another common user and developer request.
 
 I'd like to propose that we start rolling a series of monthly-ish series of
 3.0 alpha releases ASAP, with myself volunteering to take on the RM and
 other cat herding responsibilities. There are already quite a few changes
 slated for 3.0 besides the above (for instance the shell script rewrite) so
 there's already value in a 3.0 alpha, and the more time we give downstreams
 to integrate, the better.
 
 This opens up discussion about inclusion of other changes, but I'm hoping
 to freeze incompatible changes after maybe two alphas, do a beta (with no
 further incompat changes allowed), and then finally a 3.x GA. For those
 keeping track, that means a 3.x GA in about four months.
 
 I would also like to stress though that this is not intended to be a big
 bang release. For instance, it would be great if we could maintain wire
 compatibility between 2.x and 3.x, so rolling upgrades work. Keeping
 branch-2 and branch-3 similar also makes backports easier, since we're
 likely maintaining 2.x for a while yet.
 
 Please let me know any comments / concerns related to the above. If people
 are friendly to the idea, I'd like to cut a branch-3 and start working on
 the first alpha.
 
 Best,
 Andrew

Re: [VOTE] Migration from subversion to git for version control

[sanjay Radia]

+1 
sanjay
 
 On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla ka...@cloudera.com wrote:
 I have put together this proposal based on recent discussion on this topic.
 
 Please vote on the proposal. The vote runs for 7 days.
 
   1. Migrate from subversion to git for version control.
   2. Force-push to be disabled on trunk and branch-* branches. Applying
   changes from any of trunk/branch-* to any of branch-* should be through
   git cherry-pick -x.
   3. Force-push on feature-branches is allowed. Before pulling in a
   feature, the feature-branch should be rebased on latest trunk and the
   changes applied to trunk through git rebase --onto or git cherry-pick
   commit-range.
   4. Every time a feature branch is rebased on trunk, a tag that
   identifies the state before the rebase needs to be created (e.g.
   tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once
   the feature is pulled into trunk and the tags are no longer useful.
   5. The relevance/use of tags stay the same after the migration.
 
 Thanks
 Karthik
 
 PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of
 vote and will be Lazy 2/3 majority of PMC members.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Plans of moving towards JDK7 in trunk

[sanjay Radia]

On Jun 21, 2014, at 8:01 AM, Andrew Wang andrew.w...@cloudera.com wrote:

 This is why I'd like to keep my original proposal on the table: keep going
 with branch-2 in the near term, while working towards a JDK8-based Hadoop 3
 by April next year. It doesn't need to be a big bang release either. I'd be
 delighted if we could rolling upgrade from one to the other. I just didn't
 want to rule out the inclusion of some very compelling feature outright.
 Trust me though, I'd be the first person to ask about compatibility if such
 a feature does come up.


Given your above statement  on compatibility (such as rolling upgrades),  it 
should be fine for the JDK8-based-Hadoop-release to not be 3.0 and instead 
merely be 2.x? Or do you have any incompatible changes to Hadoop protocol or 
APIs in mind during the same time period?

sanjay
-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Resolved] (MAPREDUCE-4260) Use JobObject to spawn tasks on Windows

[Sanjay Radia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sanjay Radia resolved MAPREDUCE-4260.
-

Resolution: Fixed

Commit to the windows branch. Thanks Bikas.

 Use JobObject to spawn tasks on Windows
 ---

 Key: MAPREDUCE-4260
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4260
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 1.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
 Attachments: MAPREDUCE-4260.branch-1-win.1.patch, 
 MAPREDUCE-4260.branch-1-win.2.patch, MAPREDUCE-4260.branch-1-win.patch, 
 MAPREDUCE-4260.patch, test.cpp


 Currently, the Windows version spawns the task as a normal cmd shell from 
 which other downstream exe's are spawned. However, this is not bullet proof 
 because if an intermediate process exits before its child exits, then the 
 parent child process tree relationship cannot be constructed. Windows has a 
 concept of JobObject that is similar to the setsid behavior used in Linux. 
 The initial spawned task could be launched within its JobObject. Thereafter, 
 process termination, memory management etc could be operated on the JobObject.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-4204) Refactor ProcfsBasedProcessTree to make the resource collection object pluggable

[Sanjay Radia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sanjay Radia resolved MAPREDUCE-4204.
-

Resolution: Fixed

Thanks Bikas - Committed to branch-1-win

 Refactor ProcfsBasedProcessTree to make the resource collection object 
 pluggable
 

 Key: MAPREDUCE-4204
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4204
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Bikas Saha
 Attachments: MAPREDUCE-4204-1.patch, MAPREDUCE-4204.patch


 Making it a pluggable interface will allow replacing the procfs based 
 implementation with ones for other platforms.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3898) Hadoop for Windows - Interfacing with Windows to manage MR tasks

[Sanjay Radia (Created) (JIRA)]

Hadoop for Windows - Interfacing with Windows to manage MR tasks


 Key: MAPREDUCE-3898
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3898
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Sanjay Radia




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-2887) MR changes to match HADOOP-7524 (multiple RPC protocols)

[Sanjay Radia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sanjay Radia resolved MAPREDUCE-2887.
-

Resolution: Fixed

Committed as part of HADOOP-7524

 MR changes to match HADOOP-7524 (multiple RPC protocols)
 

 Key: MAPREDUCE-2887
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2887
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Sanjay Radia
Assignee: Sanjay Radia
 Fix For: 0.23.0, 0.24.0

 Attachments: rpc6ForMR.patch, rpc7ForMR.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-2887) MR changes to match HADOOP-7524 (multiple RPC protocols)

[Sanjay Radia (JIRA)]

MR changes to match HADOOP-7524 (multiple RPC protocols)


 Key: MAPREDUCE-2887
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2887
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Sanjay Radia
Assignee: Sanjay Radia




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira