Re: [VOTE] Merge yarn-native-services branch into trunk

[Sunil G]

+1 (binding)

Tested the branch code and brought up services like sleep and httpd. Also
verified UI as well.

- Sunil


On Tue, Oct 31, 2017 at 1:36 AM Jian He  wrote:

> Hi All,
>
> I would like to restart the vote for merging yarn-native-services to trunk.
> Since last vote, we have been working on several issues in documentation,
> DNS, CLI modifications etc. We believe now the feature is in a much better
> shape.
>
> Some back ground:
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate
> existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to
> deploy a service via a simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS
> service to enable users to discover services deployed on YARN via standard
> DNS lookup
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the
> existing system, and have no impact on existing system if disabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible
> without their ideas and hard work.
> Also thanks Allen for some review and verifications.
>
> Thanks,
> Jian
>
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
>

Re: [VOTE] Merge yarn-native-services branch into trunk

[Eric Yang]

+1 (binding) for branch merge.  I couldn’t get the patch to work properly 
though.  Yarn-native-services branch works fine.
If bugs surface in trunk, we will fix it.

Regards,
Eric

On 11/3/17, 2:45 PM, "Wangda Tan"  wrote:

Thanks all for the great works!

+1 (Binding). Tried to use native services to build and run applications
successfully.

- Wangda

On Fri, Nov 3, 2017 at 11:50 AM, Arun Suresh  wrote:

> +1 (binding)
>
> Cheers
> -Arun
>
> On Nov 3, 2017 11:44 AM, "Chandni Singh"  wrote:
>
> > +1
> > Thanks,
> > Chandni
> > 
> > From: Jian He 
> > Sent: Monday, October 30, 2017 1:49 PM
> > To: yarn-...@hadoop.apache.org; Hdfs-dev; Hadoop Common;
> > mapreduce-...@hadoop.apache.org
> > Subject: Re: [VOTE] Merge yarn-native-services branch into trunk
> >
> > Few more things:
> >
> > This is the document for trying a non-docker service on YARN.
> > https://github.com/apache/hadoop/blob/yarn-native-
> > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> > src/site/markdown/yarn-service/QuickStart.md
> >
> > And the document for a docker based service
> > https://github.com/apache/hadoop/blob/yarn-native-
> > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> > src/site/markdown/yarn-service/Examples.md
> >
> > And the vote lasts 7 days as usual.
> >
> > Thanks,
> > Jian
> >
> > On Oct 30, 2017, at 1:06 PM, Jian He  > e...@hortonworks.com>> wrote:
> >
> > Hi All,
> >
> > I would like to restart the vote for merging yarn-native-services to
> trunk.
> > Since last vote, we have been working on several issues in 
documentation,
> > DNS, CLI modifications etc. We believe now the feature is in a much
> better
> > shape.
> >
> > Some back ground:
> > At a high level, the following are the key feautres implemented.
> > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to
> orchestrate
> > existing services to YARN either docker or non-docker based.
> > - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to
> > deploy a service via a simple JSON spec
> > - YARN-4757[3]. Extending today's service registry with a simple DNS
> > service to enable users to discover services deployed on YARN via
> standard
> > DNS lookup
> > - YARN-6419[4]. UI support for native-services on the new YARN UI
> > All these new services are optional and are sitting outside of the
> > existing system, and have no impact on existing system if disabled.
> >
> > Special thanks to a team of folks who worked hard towards this: Billie
> > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> Sharma
> > K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible
> > without their ideas and hard work.
> > Also thanks Allen for some review and verifications.
> >
> > Thanks,
> > Jian
> >
> > [1] https://issues.apache.org/jira/browse/YARN-5079
> > [2] https://issues.apache.org/jira/browse/YARN-4793
> > [3] https://issues.apache.org/jira/browse/YARN-4757
> > [4] https://issues.apache.org/jira/browse/YARN-6419
> >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

[DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

Hi all,

With 3.0.0 GA around the corner (tx for the push, Andrew!), 2.9.0 RC out (tx 
Arun / Subru!) and 2.8.2 (tx Junping!), I think it's high time we have a 
discussion on how we manage our developmental bandwidth between 2.x line and 
3.x lines.

Once 3.0 GA goes out, we will have two parallel and major release lines. The 
last time we were in this situation was back when we did 1.x -> 2.x jump.

The parallel releases implies overhead of decisions, branch-merges and 
back-ports. Right now we already do backports for 2.7.5, 2.8.2, 2.9.1, 3.0.1 
and potentially a 3.1.0 in a few months after 3.0.0 GA. And many of these lines 
- for e.g 2.8, 2.9 - are going to be used for a while at a bunch of large 
sites! At the same time, our users won't migrate to 3.0 GA overnight - so we do 
have to support two parallel lines.

I propose we start thinking of the fate of branch-2. The idea is to have one 
final release that helps our users migrate from 2.x to 3.x. This includes any 
changes on the older line to bridge compatibility issues, upgrade issues, 
layout changes, tooling etc.

We have a few options I think
 (A)
-- Make 2.9.x the last minor release off branch-2
-- Have a maintenance release that bridges 2.9 to 3.x
-- Continue to make more maintenance releases on 2.8 and 2.9 as necessary
-- All new features obviously only go into the 3.x line as no features can 
go into the maint line.

 (B)
-- Create a new 2.10 release which doesn't have any new features, but as a 
bridging release
-- Continue to make more maintenance releases on 2.8, 2.9 and 2.10 as 
necessary
-- All new features, other than the bridging changes, go into the 3.x line

 (C)
-- Continue making branch-2 releases and postpone this discussion for later

I'm leaning towards (A) or to a lesser extent (B). Willing to hear otherwise.

Now, this obviously doesn't mean blocking of any more minor releases on 
branch-2. Obviously, any interested committer / PMC can roll up his/her 
sleeves, create a release plan and release, but we all need to acknowledge that 
versions are not cheap and figure out how the community bandwidth is split 
overall.

Thanks
+Vinod
PS: The proposal is obviously not to force everyone to go in one direction but 
more of a nudging the community to figure out if we can focus a major part of 
of our bandwidth on one line. I had a similar concern when we were doing 2.8 
and 3.0 in parallel, but the impending possibility of spreading too thin is 
much worse IMO.
PPS: (C) is a bad choice. With 2.8 and 2.9 we are already seeing user adoption 
splintering between two lines. With 2.10, 2.11 etc coexisting with 3.0, 3.1 
etc, we will revisit the mad phase years ago when we had 0.20.x, 0.20-security 
coexisting with 0.21, 0.22 etc.

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Arun Suresh]

Hey Vinod,

I've cleaned up the RC directory as you requested.

Cheers
-Arun

On Fri, Nov 3, 2017 at 4:09 PM, Vinod Kumar Vavilapalli 
wrote:

> Arun / Subru,
>
> Thanks for the great work!
>
> Few quick comments
>  - Can you cleanup the RC folder to only have tar.gz and src.tar.gz and
> their signatures and delete everything else? So that it's easy to pick up
> the important bits for the voters. For e.g, like this
> http://people.apache.org/~vinodkv/hadoop-2.8.1-RC3/
>  - Can you put the generated CHANGES.html and releasenotes.html instead of
> the md files, for quicker perusal?
>
> Thanks
> +Vinod
>
> On Nov 3, 2017, at 3:50 PM, Arun Suresh  wrote:
>
> Hi folks,
>
> Apache Hadoop 2.9.0 is the first stable release of Hadoop 2.9 line and
> will be the latest stable/production release for Apache Hadoop - it
> includes 30 New Features with 500+ subtasks, 407 Improvements, 787 Bug
> fixes new fixed issues since 2.8.2 .
>
>  More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
>  Roadmap#Roadmap-Version2.9>*
>
>  New RC is available at:
> http://home.apache.org/~asuresh/hadoop-2.9.0-RC0/
>
>  The RC tag in git is: release-2.9.0-RC0, and the latest commit id is:
> 6697f0c18b12f1bdb99cbdf81394091f4fef1f0a
>
>  The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1065/
>  >*
>
>  Please try the release and vote; the vote will run for the usual 5
> days, ending on 11/10/2017 4pm PST time.
>
> Thanks,
>
> Arun/Subru
>
>
>

Re: Cutting branch-2.9

[Subru Krishnan]

Hello Junping

Thanks for taking the time/effort to go through this. Including our reply
so that folks in the list have the full context.

> 1. It looks like we change from log4j to sel4j (HADOOP-12956) in 2.9
which is a huge incompatible change. Do we have consensus on this in
community (as I saw HBase community are deny this kind of change in minor
release)? If so, we should document somewhere what is changed and what's
the impact.
As per the compatibility guidelines (http://hadoop.apache.org/
docs/current/hadoop-project-dist/hadoop-common/Compatibility.html#Log4j_
Configuration_Files), our understanding is that if SLF4j can read the old
Log4j property files, we should be good - and it is the case.

>  2. Some APIs (deprecated in 2.8) get removed. My understanding is we
only remove deprecated API in next major release (3.0) but not minor
release. Please double check this with some documents or other guys for
certain answer.
We see only 2 DEPRECATED methods have been removed:
- The AllocateResponse.newInstance method and the RMProxy.createRMProxy
methods. Both are internal builders and are not really APIs IMHO - so we
feel this should not be a blocker.

>  3. Some stable APIs get removed, like: YarnScheduler.allocate(), etc. We
may need to add it back before we go for RC stage.
Are you sure you are running JACC against latest 2.8.2 artifacts - as we
saw a much smaller delta when we ran jdiff ? Case in point: Looks like the
YarnScheduler#allocate() changes were already part of 2.8.2 as per (
https://github.com/apache/hadoop/blob/branch-2.8.2/
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-
resourcemanager/src/main/java/org/apache/hadoop/yarn/server/
resourcemanager/scheduler/YarnScheduler.java) and the relevant JIRA is
YARN-5221.

-Subru/Arun

On Fri, Nov 3, 2017 at 4:24 PM, Junping Du  wrote:

> Below is some findings from my recently run of JACC (
> https://builds.apache.org/job/Hadoop-2.9-JACC/12/
> artifact/target/compat-check/report.html) job against 2.9 and 2.8.2. I
> have discussed with Arun and Subru offline on jdiff report who convinced me
> some of items are not a big concern. Just tried to list here in case people
> in community have different voices:
>
>  My original concerns of incompatibilities:
>  1. It looks like we change from log4j to sel4j (HADOOP-12956) in 2.9
> which is a huge incompatible change. Do we have consensus on this in
> community - as I saw HBase community are deny this kind of change in minor
> release? I talked with several other guys who seems to have different
> opinion.
>
> 2. Some APIs (deprecated in 2.8) get removed. My understanding is we
> only remove deprecated API in next major release (3.0) but not minor
> release. - Arun and Subru convinced me that removed 2 methods are not
> supposed to be used by other downstream projects
>
> 3. Some stable APIs get removed, like: YarnScheduler.allocate(), etc.
> We may need to add it back for compatibility of Scheduler API.
>
> Thoughts?
>
> Thanks,
>
> Junping
>
> 
> From: Arun Suresh 
> Sent: Thursday, November 2, 2017 11:55 PM
> To: 俊平堵
> Cc: Arun Suresh; su...@apache.org; common-dev@hadoop.apache.org;
> yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org
> Subject: Re: Cutting branch-2.9
>
> Quick update folks.
>
> We just cut branch-2.9.0, and set the mvn version to 2.9.0
> We've also updated the version on branch-2.9 to 2.9.1-SNAPSHOT.
>
> Expect an RC in the next 24 hrs :)
>
> Cheers
> -Arun/Subru
>
> On Tue, Oct 31, 2017 at 5:24 PM, Arun Suresh 
> wrote:
>
> > Hi Junping,
> >
> > > In the mean time, branch-2.9 should be reserved for next 2.9 point
> > release (2.9.1) and branch-2 should be reserved for next minor release
> > (2.10.0 or whatever name it is). Thoughts?
> > Yup, That is the current plan. We've updated mvn versions in branch-2 to
> > point to 2.10.0-SNAPSHOT.
> > Once we cut branch-2.9.0, we will update mvn versions on branch-2.9.0 to
> > 2.9.0 and set versions on branch-2.9 to 2.9.1-SNAPSHOT (We plan to do
> that
> > by Thursday)
> >
> > Cheers
> > -Arun
> >
> > On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵  wrote:
> >
> >> Hi Arun/Subru,
> >> Thanks for updating on 2.9.0 release progress. A quick question
> here:
> >> are we planning to release from branch-2.9 directly?
> >> I doubt this as it seems against our current branch release
> practice (
> >> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of
> >> any confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release
> >> work. In the mean time, branch-2.9 should be reserved for next 2.9 point
> >> release (2.9.1) and branch-2 should be reserved for next minor release
> >> (2.10.0 or whatever name it is). Thoughts?
> >>
> >> bq. @Junping, lets move the jdiff conversation to separate thread.
> >> Sure. I will reply jdiff in 

Re: Cutting branch-2.9

[Junping Du]

Below is some findings from my recently run of JACC 
(https://builds.apache.org/job/Hadoop-2.9-JACC/12/artifact/target/compat-check/report.html)
 job against 2.9 and 2.8.2. I have discussed with Arun and Subru offline on 
jdiff report who convinced me some of items are not a big concern. Just tried 
to list here in case people in community have different voices:

 My original concerns of incompatibilities:
 1. It looks like we change from log4j to sel4j (HADOOP-12956) in 2.9 which 
is a huge incompatible change. Do we have consensus on this in community - as I 
saw HBase community are deny this kind of change in minor release? I talked 
with several other guys who seems to have different opinion.

2. Some APIs (deprecated in 2.8) get removed. My understanding is we only 
remove deprecated API in next major release (3.0) but not minor release. - Arun 
and Subru convinced me that removed 2 methods are not supposed to be used by 
other downstream projects

3. Some stable APIs get removed, like: YarnScheduler.allocate(), etc. We 
may need to add it back for compatibility of Scheduler API.

Thoughts?

Thanks,

Junping


From: Arun Suresh 
Sent: Thursday, November 2, 2017 11:55 PM
To: 俊平堵
Cc: Arun Suresh; su...@apache.org; common-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org
Subject: Re: Cutting branch-2.9

Quick update folks.

We just cut branch-2.9.0, and set the mvn version to 2.9.0
We've also updated the version on branch-2.9 to 2.9.1-SNAPSHOT.

Expect an RC in the next 24 hrs :)

Cheers
-Arun/Subru

On Tue, Oct 31, 2017 at 5:24 PM, Arun Suresh  wrote:

> Hi Junping,
>
> > In the mean time, branch-2.9 should be reserved for next 2.9 point
> release (2.9.1) and branch-2 should be reserved for next minor release
> (2.10.0 or whatever name it is). Thoughts?
> Yup, That is the current plan. We've updated mvn versions in branch-2 to
> point to 2.10.0-SNAPSHOT.
> Once we cut branch-2.9.0, we will update mvn versions on branch-2.9.0 to
> 2.9.0 and set versions on branch-2.9 to 2.9.1-SNAPSHOT (We plan to do that
> by Thursday)
>
> Cheers
> -Arun
>
> On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵  wrote:
>
>> Hi Arun/Subru,
>> Thanks for updating on 2.9.0 release progress. A quick question here:
>> are we planning to release from branch-2.9 directly?
>> I doubt this as it seems against our current branch release practice (
>> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of
>> any confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release
>> work. In the mean time, branch-2.9 should be reserved for next 2.9 point
>> release (2.9.1) and branch-2 should be reserved for next minor release
>> (2.10.0 or whatever name it is). Thoughts?
>>
>> bq. @Junping, lets move the jdiff conversation to separate thread.
>> Sure. I will reply jdiff in separated thread.
>>
>> Thanks,
>>
>> Junping
>>
>> 2017-10-31 13:44 GMT-07:00 Arun Suresh :
>>
>>> Hello folks
>>>
>>> We just cut branch-2.9 since all the critical/blocker issues are now
>>> resolved.
>>> We plan to perform some sanity checks for the rest of the week and cut
>>> branch-2.9.0 and push out an RC0 by the end of the week.
>>>
>>> Kindly refrain from committing to branch-2.9 without giving us a heads
>>> up.
>>>
>>> @Junping, lets move the jdiff conversation to separate thread.
>>>
>>> Cheers
>>> -Arun/Subru
>>>
>>> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan 
>>> wrote:
>>>
>>> > We want to give heads up that we are going to cut branch-2.9 tomorrow
>>> > morning.
>>> >
>>> > We are down to 3 blockers and they all are close to being committed
>>> > (thanks everyone):
>>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>>> >
>>> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which
>>> are
>>> > close to completion.
>>> >
>>> > Folks who are working/reviewing these, kindly prioritize accordingly so
>>> > that we can make the release on time.
>>> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>>> >
>>> > Thanks in advance!
>>> >
>>> > -Subru/Arun
>>> >
>>> >
>>> >
>>>
>>
>>
>


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Vinod Kumar Vavilapalli]

Arun / Subru,

Thanks for the great work!

Few quick comments
 - Can you cleanup the RC folder to only have tar.gz and src.tar.gz and their 
signatures and delete everything else? So that it's easy to pick up the 
important bits for the voters. For e.g, like this 
http://people.apache.org/~vinodkv/hadoop-2.8.1-RC3/ 

 - Can you put the generated CHANGES.html and releasenotes.html instead of the 
md files, for quicker perusal?

Thanks
+Vinod

> On Nov 3, 2017, at 3:50 PM, Arun Suresh  wrote:
> 
> Hi folks,
> 
> Apache Hadoop 2.9.0 is the first stable release of Hadoop 2.9 line and
> will be the latest stable/production release for Apache Hadoop - it
> includes 30 New Features with 500+ subtasks, 407 Improvements, 787 Bug
> fixes new fixed issues since 2.8.2 .
> 
>  More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
> 
>  New RC is available at:
> http://home.apache.org/~asuresh/hadoop-2.9.0-RC0/
> 
>  The RC tag in git is: release-2.9.0-RC0, and the latest commit id is:
> 6697f0c18b12f1bdb99cbdf81394091f4fef1f0a
> 
>  The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1065/
> *
> 
>  Please try the release and vote; the vote will run for the usual 5
> days, ending on 11/10/2017 4pm PST time.
> 
> Thanks,
> 
> Arun/Subru

[VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Arun Suresh]

Hi folks,

 Apache Hadoop 2.9.0 is the first stable release of Hadoop 2.9 line and
will be the latest stable/production release for Apache Hadoop - it
includes 30 New Features with 500+ subtasks, 407 Improvements, 787 Bug
fixes new fixed issues since 2.8.2 .

  More information about the 2.9.0 release plan can be found here:
*https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
*

  New RC is available at:
http://home.apache.org/~asuresh/hadoop-2.9.0-RC0/

  The RC tag in git is: release-2.9.0-RC0, and the latest commit id is:
6697f0c18b12f1bdb99cbdf81394091f4fef1f0a

  The maven artifacts are available via repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1065/
*

  Please try the release and vote; the vote will run for the usual 5
days, ending on 11/10/2017 4pm PST time.

Thanks,

Arun/Subru

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Vinod Kumar Vavilapalli]

>   At a minimum, it should at least be using it’s own maven module for a 
> lot of the bits that generates it’s own maven jars so that we can split this 
> functionality up at build/test time.


I expected this to be the case, but looks like it isn't.

There's lot of value in splitting the HDFS code into smaller modules. 
Definitely newer code like Ozone.

When we did this for YARN, initially there were concerns about module 
proliferation, but looking back, my observations have been that it has done us 
far more good than expected. Starting with the fact that we had clients and 
servers modularized independently, as well as servers from other servers, with 
far cleaner contracts than what we had in Hadoop 1 world.

Thanks
+Vinod

Re: [VOTE] Merge yarn-native-services branch into trunk

[Wangda Tan]

Thanks all for the great works!

+1 (Binding). Tried to use native services to build and run applications
successfully.

- Wangda

On Fri, Nov 3, 2017 at 11:50 AM, Arun Suresh  wrote:

> +1 (binding)
>
> Cheers
> -Arun
>
> On Nov 3, 2017 11:44 AM, "Chandni Singh"  wrote:
>
> > +1
> > Thanks,
> > Chandni
> > 
> > From: Jian He 
> > Sent: Monday, October 30, 2017 1:49 PM
> > To: yarn-...@hadoop.apache.org; Hdfs-dev; Hadoop Common;
> > mapreduce-...@hadoop.apache.org
> > Subject: Re: [VOTE] Merge yarn-native-services branch into trunk
> >
> > Few more things:
> >
> > This is the document for trying a non-docker service on YARN.
> > https://github.com/apache/hadoop/blob/yarn-native-
> > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> > src/site/markdown/yarn-service/QuickStart.md
> >
> > And the document for a docker based service
> > https://github.com/apache/hadoop/blob/yarn-native-
> > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> > src/site/markdown/yarn-service/Examples.md
> >
> > And the vote lasts 7 days as usual.
> >
> > Thanks,
> > Jian
> >
> > On Oct 30, 2017, at 1:06 PM, Jian He  > e...@hortonworks.com>> wrote:
> >
> > Hi All,
> >
> > I would like to restart the vote for merging yarn-native-services to
> trunk.
> > Since last vote, we have been working on several issues in documentation,
> > DNS, CLI modifications etc. We believe now the feature is in a much
> better
> > shape.
> >
> > Some back ground:
> > At a high level, the following are the key feautres implemented.
> > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to
> orchestrate
> > existing services to YARN either docker or non-docker based.
> > - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to
> > deploy a service via a simple JSON spec
> > - YARN-4757[3]. Extending today's service registry with a simple DNS
> > service to enable users to discover services deployed on YARN via
> standard
> > DNS lookup
> > - YARN-6419[4]. UI support for native-services on the new YARN UI
> > All these new services are optional and are sitting outside of the
> > existing system, and have no impact on existing system if disabled.
> >
> > Special thanks to a team of folks who worked hard towards this: Billie
> > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> Sharma
> > K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible
> > without their ideas and hard work.
> > Also thanks Allen for some review and verifications.
> >
> > Thanks,
> > Jian
> >
> > [1] https://issues.apache.org/jira/browse/YARN-5079
> > [2] https://issues.apache.org/jira/browse/YARN-4793
> > [3] https://issues.apache.org/jira/browse/YARN-4757
> > [4] https://issues.apache.org/jira/browse/YARN-6419
> >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Ravi Prakash]

Hi folks!

Thank you for sharing the design docs and the tremendous amount of work
that has gone into Ozone. I'm grateful that atleast someone is trying to
drastically improve HDFS.

*If* there is a meeting to discuss this merge, could I please also be
invited?

Have we ever thought about distributing the Namenode metadata across nodes
dynamically based on load and RPC times (unlike static federation that we
have now)?

Also, I think a major feature that HDFS still lacks (and a lot of our users
ask for) is BCP / Disaster Recovery. I only bring this up to see if the
choice of proposed design would have implications for that later on.

Thanks,
Ravi

On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia  wrote:

> Konstantine,
>  Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
>  that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> > On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko 
> wrote:
> >
> > Hey guys,
> >
> > It is an interesting question whether Ozone should be a part of Hadoop.
> > There are two main reasons why I think it should not.
> >
> > 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> > sizable community behind, it looks to me like a whole new project.
> > It is essentially a new storage system, with different (than HDFS)
> > architecture, separate S3-like APIs. This is really great - the World
> sure
> > needs more distributed file systems. But it is not clear why Ozone should
> > co-exist with HDFS under the same roof.
> >
> > 2. Ozone is probably just the first step in rebuilding HDFS under a new
> > architecture. With the next steps presumably being HDFS-10419 and
> > HDFS-8.
> > The design doc for the new architecture has never been published. I can
> > only assume based on some presentations and personal communications that
> > the idea is to use Ozone as a block storage, and re-implement NameNode,
> so
> > that it stores only a partial namesapce in memory, while the bulk of it
> > (cold data) is persisted to a local storage.
> > Such architecture makes me wonder if it solves Hadoop's main problems.
> > There are two main limitations in HDFS:
> >  a. The throughput of Namespace operations. Which is limited by the
> number
> > of RPCs the NameNode can handle
> >  b. The number of objects (files + blocks) the system can maintain. Which
> > is limited by the memory size of the NameNode.
> > The RPC performance (a) is more important for Hadoop scalability than the
> > object count (b). The read RPCs being the main priority.
> > The new architecture targets the object count problem, but in the expense
> > of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> > Also based on the use patterns on our large clusters we read up to 90% of
> > the data we write, so cold data is a small fraction and most of it must
> be
> > cached.
> >
> > To summarize:
> > - Ozone is a big enough system to deserve its own project.
> > - The architecture that Ozone leads to does not seem to solve the
> intrinsic
> > problems of current HDFS.
> >
> > I will post my opinion in the Ozone jira. Should be more convenient to
> > discuss it there for further reference.
> >
> > Thanks,
> > --Konstantin
> >
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei 
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >>  1.  All services, management scripts
> >>  2.  Object store APIs, exposed via both REST and RPC
> >>  3.  Master service UIs, command line interfaces
> >>  4.  Pluggable pipeline Integration
> >>  5.  Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >>  6.  Corona - a load generator for Ozone.
> >>  7.  Essential documentation added to Hadoop site.
> >>  8.  Version specific Ozone Documentation, accessible via service UI.
> >>  9.  Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confl
> >> uence/display/HADOOP/Dev+cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ 

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[sanjay Radia]

Konstantine, 
 Thanks for your comments, questions and feedback. I have attached a document 
to the HDFS-7240 jira 
 that explains a design for scaling HDFS and how Ozone paves the way towards 
the full solution.


https://issues.apache.org/jira/secure/attachment/12895963/HDFS%20Scalability%20and%20Ozone.pdf


sanjay




> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko  wrote:
> 
> Hey guys,
> 
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
> 
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
> 
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-8.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
>  a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
>  b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
> 
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
> 
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
> 
> Thanks,
> --Konstantin
> 
> 
> 
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei  wrote:
> 
>> Hello everyone,
>> 
>> 
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>> 
>> 
>> 
>> The merge payload includes the following:
>> 
>>  1.  All services, management scripts
>>  2.  Object store APIs, exposed via both REST and RPC
>>  3.  Master service UIs, command line interfaces
>>  4.  Pluggable pipeline Integration
>>  5.  Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>>  6.  Corona - a load generator for Ozone.
>>  7.  Essential documentation added to Hadoop site.
>>  8.  Version specific Ozone Documentation, accessible via service UI.
>>  9.  Docker support for ozone, which enables faster development cycles.
>> 
>> 
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confl
>> uence/display/HADOOP/Dev+cluster+with+docker.
>> 
>> 
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>> 
>> 
>> Please share your thoughts, thanks!
>> 
>> 
>> -- Weiwei Yang
>> 
> 
> 
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei  wrote:
> 
>> Hello everyone,
>> 
>> 
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>> 
>> 
>> 
>> The merge payload includes the following:
>> 
>>  1.  All services, management scripts
>>  2.  Object store APIs, exposed via both REST and RPC
>>  3.  Master service UIs, command line interfaces
>>  4.  Pluggable pipeline Integration

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Allen Wittenauer]

> On Nov 3, 2017, at 12:08 PM, Stack  wrote:
> 
> On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko 
> wrote:
> 
>> It is an interesting question whether Ozone should be a part of Hadoop.
> 
> I don't see a direct answer to this question. Is there one? Pardon me if
> I've not seen it but I'm interested in the response.

+1

Given:

* a completely different set of config files (ozone-site.xml, etc)
* package name is org.apache.hadoop.ozone, not 
org.apache.hadoop.hdfs.ozone

… it doesn’t really seem to want to be part of HDFS, much less Hadoop.

Plus hadoop-hdfs-project/hadoop-hdfs is already a battle zone when it comes to 
unit tests, dependencies, etc [*]

At a minimum, it should at least be using it’s own maven module for a 
lot of the bits that generates it’s own maven jars so that we can split this 
functionality up at build/test time.

At a higher level, this feels a lot like the design decisions that were 
made around yarn-native-services.  This feature is either part of HDFS or it’s 
not. Pick one.  Doing both is incredibly confusing for everyone outside of the 
branch.
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Xiaoyu Yao]

Hi Lei,

Thank you for your interest in Ozone. Let me answer each of the
specific questions.

> As the current state of Ozone implementation, what are the major
> benefits of using today’s Ozone over HDFS?


Scale -  HDFS tops out at 500/700 million keys; Ozone primary 
use case is to go beyond that limit. Ozone can scale block space 
and namespace independently. Ozone also provides object store 
semantics which is simpler and scales better


Enabling new workloads on HDFS -  For example, we see lots of 
customers moving to a cloud-like model, where they have a compute 
cluster and storage cluster. This allows them to move workloads to 
the cloud and back seamlessly. We see a marked increase of 
Docker/Kubernetes enabled workloads. The pain point for most 
Docker/VM deployments is lack of storage. Ozone is an excellent 
fit for Docker/Kubernetes deployments.

Ease of Management and use - Ozone has learned very 
valuable lessons from HDFS. It comes with a good set of 
management tools and tries to avoid very complicated setup.


> Giving that its missing features like HDFS-12680 and HDFS-12697, and
> the closing of Hadoop 3.0 release, should we wait for a late merge
> when Ozone is more mature ?

Both HDFS-12680 (lease manager) and HDFS-12697 (Ozone services 
stay disabled in secure setup) are resolved in the past weeks. 
We are targeting the merge for trunk not 3.0.

> Or more generally, why should this merge to a release branch happen
> now, when Ozone is not yet usable by users? Staying on a feature
> branch seems like it's still the right place to Me.

Let me repeat that we are not merging to the 3.0 branch. We are
merging to trunk and we do not intend to backport this to 3.0 code
base.

Ozone is certainly usable. We have written and read billions of keys
into ozone. I would think that it more like Erasure coding when we
merged. We want ozone to be used/tested when people start using 3.1
release. Yes, it is an Alpha feature, having an Alpha release out in
the community is the best way to mature Ozone.


> For the existing HDFS user, could you address the semantic gaps
> between Ozone / Ozone File System and HDFS.

Ozone file system offers a Hadoop compatible file system. For the
first release, we are targeting YARN, Hive, and Spark as the principle
workloads. These applications are functional with Ozone file system.

> It would be great to illustrate what is the expected use cases for
> Ozone giving its different architecture and design decisions?

We expect almost all real use case of ozone to come via Ozone File
System. Hence our focus has been to make sure that (YARN, Hive and
Spark) work well with this system. Ozone file system does the right
magic on behalf of the users for now.

> Like no append, no atomic rename and etc.

This is similar to S3 -- the rise of cloud-based object stores has made 
it very easy for ozone. In fact, the work done by other stacks (Hive, Spark 
etc.) 
to enable big data workload in cloud is extremely helpful for ozone.


> A follow question, was it able to run any of today’s Hadoop
> applications (MR, Spark, Impala, Presto and etc) on Ozone directly, or
> against OZoneFileSystem? I think a performance / scalability gain or
> extended functionality should be the prerequisites for the merge.
> Additionally, I believe such tests will reveal the potential caveats
> if any.

We have run, Mapreduce (pretty much all standard applications along
with Distcp works well), YARN, Hive and Spark against ozone with NO
Modifications to MR,YARN, Hive or Spark.

We have never tried out Impala or Presto, but if they are known to work
well against Hadoop compatible file systems, I am hopeful that they
will work as well. Please feel free to test and report if you run into
any issues.


> * Ozone’s architecture shows great potential to address NN
> scalability.  However it looks like a XXL effort to me, considering
> the fact that 1) the community had multiple unfinished attempts to
> simply separate namespace and block management within the same NN
> process,

You are absolutely correct. We have learned from those experiences. We
think that separating namespace and block space in the same NN process
does not address the core issue of NN scale. And, also as you clearly 
mentioned they are unfinished. 

With Ozone, we have separated out a block service. Once it is known 
to be stable, we will use that in Namenode, thus achieving the full 
separation. Ozone FS and Ozone object store are intermediate steps
 to solving the scale issue for HDFS.

> *and 2) many existing features like snapshot, append, erasure coding,
> and etc, are not straightforward to be implemented in today’s ozone
> design. Could you share your opinions on this matter?

Ozone is well prepared to implement each of these features. We have
many design documents for ozone posted in the sub-JIRAs. For example,
please take a look at the versioning doc to understand how Ozone’s block
layer really offers.

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Stack]

On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko 
wrote:

> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
>


I don't see a direct answer to this question. Is there one? Pardon me if
I've not seen it but I'm interested in the response.

I ask because IMO the "Hadoop" project is over-stuffed already. Just see
the length of the cc list on this email. Ozone could be standalone. It is a
coherent enough effort.

Thanks,
St.Ack





> There are two main reasons why I think it should not.
>
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-8.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
>   a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
>   b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei 
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> >   1.  All services, management scripts
> >   2.  Object store APIs, exposed via both REST and RPC
> >   3.  Master service UIs, command line interfaces
> >   4.  Pluggable pipeline Integration
> >   5.  Ozone File System (Hadoop compatible file system implementation,
> > passes all FileSystem contract tests)
> >   6.  Corona - a load generator for Ozone.
> >   7.  Essential documentation added to Hadoop site.
> >   8.  Version specific Ozone Documentation, accessible via service UI.
> >   9.  Docker support for ozone, which enables faster development cycles.
> >
> >
> > To build Ozone and run ozone using docker, please follow instructions in
> > this wiki page. https://cwiki.apache.org/confl
> > uence/display/HADOOP/Dev+cluster+with+docker.
> >
> >
> > We have built a passionate and diverse community to drive this feature
> > development. As a team, we have achieved significant progress in past 3
> > years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> > have resolved almost 400 JIRAs by 20+ contributors/committers from
> > different countries and affiliations. We also want to thank the large
> > number of community members who were supportive of our efforts and
> > contributed ideas and participated in the design of ozone.
> >
> >
> > Please share your thoughts, thanks!
> >
> >
> > -- Weiwei Yang
> >
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei 
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> >   1.  All services, management scripts
> >   2.  Object store APIs, exposed via both REST and RPC
> >   3.  Master service UIs, command line 

Re: [VOTE] Merge yarn-native-services branch into trunk

[Arun Suresh]

+1 (binding)

Cheers
-Arun

On Nov 3, 2017 11:44 AM, "Chandni Singh"  wrote:

> +1
> Thanks,
> Chandni
> 
> From: Jian He 
> Sent: Monday, October 30, 2017 1:49 PM
> To: yarn-...@hadoop.apache.org; Hdfs-dev; Hadoop Common;
> mapreduce-...@hadoop.apache.org
> Subject: Re: [VOTE] Merge yarn-native-services branch into trunk
>
> Few more things:
>
> This is the document for trying a non-docker service on YARN.
> https://github.com/apache/hadoop/blob/yarn-native-
> services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> src/site/markdown/yarn-service/QuickStart.md
>
> And the document for a docker based service
> https://github.com/apache/hadoop/blob/yarn-native-
> services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> src/site/markdown/yarn-service/Examples.md
>
> And the vote lasts 7 days as usual.
>
> Thanks,
> Jian
>
> On Oct 30, 2017, at 1:06 PM, Jian He  e...@hortonworks.com>> wrote:
>
> Hi All,
>
> I would like to restart the vote for merging yarn-native-services to trunk.
> Since last vote, we have been working on several issues in documentation,
> DNS, CLI modifications etc. We believe now the feature is in a much better
> shape.
>
> Some back ground:
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate
> existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to
> deploy a service via a simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS
> service to enable users to discover services deployed on YARN via standard
> DNS lookup
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the
> existing system, and have no impact on existing system if disabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible
> without their ideas and hard work.
> Also thanks Allen for some review and verifications.
>
> Thanks,
> Jian
>
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
>
>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Merge yarn-native-services branch into trunk

[Chandni Singh]

+1 
Thanks,
Chandni

From: Jian He 
Sent: Monday, October 30, 2017 1:49 PM
To: yarn-...@hadoop.apache.org; Hdfs-dev; Hadoop Common; 
mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Merge yarn-native-services branch into trunk

Few more things:

This is the document for trying a non-docker service on YARN.
https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/QuickStart.md

And the document for a docker based service
https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/Examples.md

And the vote lasts 7 days as usual.

Thanks,
Jian

On Oct 30, 2017, at 1:06 PM, Jian He 
> wrote:

Hi All,

I would like to restart the vote for merging yarn-native-services to trunk.
Since last vote, we have been working on several issues in documentation, DNS, 
CLI modifications etc. We believe now the feature is in a much better shape.

Some back ground:
At a high level, the following are the key feautres implemented.
- YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
existing services to YARN either docker or non-docker based.
- YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to deploy 
a service via a simple JSON spec
- YARN-4757[3]. Extending today's service registry with a simple DNS service to 
enable users to discover services deployed on YARN via standard DNS lookup
- YARN-6419[4]. UI support for native-services on the new YARN UI
All these new services are optional and are sitting outside of the existing 
system, and have no impact on existing system if disabled.

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, 
Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, 
Akhil PB, Eric Yang. This effort could not be possible without their ideas and 
hard work.
Also thanks Allen for some review and verifications.

Thanks,
Jian

[1] https://issues.apache.org/jira/browse/YARN-5079
[2] https://issues.apache.org/jira/browse/YARN-4793
[3] https://issues.apache.org/jira/browse/YARN-4757
[4] https://issues.apache.org/jira/browse/YARN-6419




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/

[Nov 3, 2017 12:15:33 AM] (Arun Suresh) HADOOP-15013. Fix ResourceEstimator 
findbugs issues. (asuresh)
[Nov 3, 2017 12:39:23 AM] (subru) YARN-7432. Fix DominantResourceFairnessPolicy 
serializable findbugs
[Nov 3, 2017 1:55:29 AM] (sunilg) YARN-7410. Cleanup FixedValueResource to 
avoid dependency to
[Nov 3, 2017 4:27:35 AM] (xiao) HDFS-12682. ECAdmin -listPolicies will always 
show
[Nov 3, 2017 4:29:53 AM] (inigoiri) YARN-7434. Router getApps REST invocation 
fails with multiple RMs.
[Nov 3, 2017 4:53:13 AM] (xiao) HDFS-12725. 
BlockPlacementPolicyRackFaultTolerant fails with very uneven
[Nov 3, 2017 6:15:50 AM] (sunilg) YARN-7392. Render cluster information on new 
YARN web ui. Contributed by




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-hdfs:5 
   hadoop-mapreduce-client-jobclient:1 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 
   hadoop.hdfs.TestDecommission 
   hadoop.hdfs.server.mover.TestMover 
   
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
 
   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 
   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.TestReadStripedFileWithDNFailure 
   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.sls.TestSLSRunner 

Timed out junit tests :

   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA 
   org.apache.hadoop.mapred.pipes.TestPipeApplication 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-compile-javac-root.txt
  [280K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/whitespace-eol.txt
  [8.8M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/whitespace-tabs.txt
  [292K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   UnreapedProcessesLog:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-reaper.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-reaper.txt
  [4.0K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [1.2M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [8.0K]

   

[jira] [Created] (HADOOP-15016) Add reservation support to RPC FairCallQueue

[Wei Yan (JIRA)]

Wei Yan created HADOOP-15016:


 Summary: Add reservation support to RPC FairCallQueue
 Key: HADOOP-15016
 URL: https://issues.apache.org/jira/browse/HADOOP-15016
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Wei Yan
Assignee: Wei Yan
Priority: Normal


FairCallQueue is introduced to provide RPC resource fairness among different 
users. In current implementation, each user is weighted equally, and the 
processing priority for different RPC calls are based on how many requests that 
user sent before. This works well when the cluster is shared among several 
end-users.

However, this has some limitations when a cluster is shared among both 
end-users and some service jobs, like some ETL jobs which run under a service 
account and need to issue lots of RPC calls. When NameNode becomes quite busy, 
this set of jobs can be easily backoffed and low-prioritied. We cannot simply 
treat this type jobs as "bad" user who randomly issues too many calls, as their 
calls are normal calls. Also, it is unfair to weight a end-user and a heavy 
service user equally when allocating RPC resources.

One idea here is to introduce reservation support to RPC resources. That is, 
for some services, we reserve some RPC resources for their calls. This idea is 
very similar to how YARN manages CPU/memory resources among different resource 
queues. A little more details here: Along with existing FairCallQueue setup 
(like using 4 queues with different priorities), we would add some additional 
special queues, one for each special service user. For each special service 
user, we provide a guarantee RPC share (like 10% which can be aligned with its 
YARN resource share), and this percentage can be converted to a weight used in 
WeightedRoundRobinMultiplexer. A quick example, we have 4 default queues with 
default weights (8, 4, 2, 1), and two special service users (user1 with 10% 
share, and user2 with 15% share). So finally we'll have 6 queues, 4 default 
queues (with weights 8, 4, 2, 1) and 2 special queues (user1Queue weighted 
15*10%/75%=2, and user2Queue weighted 15*15%/75%=3).

For new coming RPC calls from special service users, they will be put directly 
to the corresponding reserved queue; for other calls, just follow current 
implementation.

By default, there is no special user and all RPC requests follow existing 
FairCallQueue implementation.

Would like to hear more comments on this approach; also want to know any other 
better solutions?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15015) TestConfigurationFieldsBase to use SLF4J for logging

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15015:
---

 Summary: TestConfigurationFieldsBase to use SLF4J for logging
 Key: HADOOP-15015
 URL: https://issues.apache.org/jira/browse/HADOOP-15015
 Project: Hadoop Common
  Issue Type: Bug
  Components: conf, test
Affects Versions: 3.0.0
Reporter: Steve Loughran


{{TestConfigurationFieldsBase}} has a protected "configDebug" field used to 
turn logging on/off

{code}
  if (configDebug) {
System.out.println("Field: " + f);
  }
{code}

Presumably its there to allow people with code access to debug their classes. 
But if we switch to SLF4J you get controllable logging @ runtime.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15014) KMS should log the IP address of the clients

[Zsombor Gegesy (JIRA)]

Zsombor Gegesy created HADOOP-15014:
---

 Summary: KMS should log the IP address of the clients
 Key: HADOOP-15014
 URL: https://issues.apache.org/jira/browse/HADOOP-15014
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 2.8.1
Reporter: Zsombor Gegesy
Priority: Major


Currently KMSMDCFilter only captures http request url and method, but not the 
remote address of the client.
 Storing this information in a thread local variable would help external 
authorizer plugins to do more thorough checks. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-9864) Adopt SLF4Js over commons-logging

[Andras Bokor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Bokor resolved HADOOP-9864.
--
Resolution: Duplicate

Adopting SLF4J is in progress (or maybe ready?) and it seems the adopting 
process is not tracked here. This one can be closed.

> Adopt SLF4Js over commons-logging
> -
>
> Key: HADOOP-9864
> URL: https://issues.apache.org/jira/browse/HADOOP-9864
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Steve Loughran
>Priority: Major
>
> This is fairly major, but it's something to raise. Commons-logging is used as 
> frozen front end to log4j with a pre-java5-varargs syntax, forcing us to wrap 
> every log event with an {{if (log.isDebugEnabled()}} clause.
> SLF4J
> # is the new de-facto standard Java logging API
> # does use varags for on-demand stringification {{log.info("routing to {}
> , host)}}
> # bridges to Log4J
> # hooks up direct to logback, which has a reputation for speed through less 
> lock contention
> # still supports the same {{isDebugEnabled()}} probes, so commons-logging 
> based classes could switch to SLF4J merely by changing the type of the 
> {{LOG}} class.
> Hadoop already depends on SLF4J for jetty support, hadoop-auth uses it 
> directly.
> This JIRA merely proposes making a decision on whether to adopt SL4J -and if 
> so, how to roll it out.
> The least-disruptive roll-out strategy would be to mandate it on new modules, 
> then switch module-by-module in the existing code.
> We'd also need to find all those tests that dig down to log4j directly, and 
> make sure that they can migrate to the new APIs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Cutting branch-2.9

[Arun Suresh]

Quick update folks.

We just cut branch-2.9.0, and set the mvn version to 2.9.0
We've also updated the version on branch-2.9 to 2.9.1-SNAPSHOT.

Expect an RC in the next 24 hrs :)

Cheers
-Arun/Subru

On Tue, Oct 31, 2017 at 5:24 PM, Arun Suresh  wrote:

> Hi Junping,
>
> > In the mean time, branch-2.9 should be reserved for next 2.9 point
> release (2.9.1) and branch-2 should be reserved for next minor release
> (2.10.0 or whatever name it is). Thoughts?
> Yup, That is the current plan. We've updated mvn versions in branch-2 to
> point to 2.10.0-SNAPSHOT.
> Once we cut branch-2.9.0, we will update mvn versions on branch-2.9.0 to
> 2.9.0 and set versions on branch-2.9 to 2.9.1-SNAPSHOT (We plan to do that
> by Thursday)
>
> Cheers
> -Arun
>
> On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵  wrote:
>
>> Hi Arun/Subru,
>> Thanks for updating on 2.9.0 release progress. A quick question here:
>> are we planning to release from branch-2.9 directly?
>> I doubt this as it seems against our current branch release practice (
>> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of
>> any confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release
>> work. In the mean time, branch-2.9 should be reserved for next 2.9 point
>> release (2.9.1) and branch-2 should be reserved for next minor release
>> (2.10.0 or whatever name it is). Thoughts?
>>
>> bq. @Junping, lets move the jdiff conversation to separate thread.
>> Sure. I will reply jdiff in separated thread.
>>
>> Thanks,
>>
>> Junping
>>
>> 2017-10-31 13:44 GMT-07:00 Arun Suresh :
>>
>>> Hello folks
>>>
>>> We just cut branch-2.9 since all the critical/blocker issues are now
>>> resolved.
>>> We plan to perform some sanity checks for the rest of the week and cut
>>> branch-2.9.0 and push out an RC0 by the end of the week.
>>>
>>> Kindly refrain from committing to branch-2.9 without giving us a heads
>>> up.
>>>
>>> @Junping, lets move the jdiff conversation to separate thread.
>>>
>>> Cheers
>>> -Arun/Subru
>>>
>>> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan 
>>> wrote:
>>>
>>> > We want to give heads up that we are going to cut branch-2.9 tomorrow
>>> > morning.
>>> >
>>> > We are down to 3 blockers and they all are close to being committed
>>> > (thanks everyone):
>>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>>> >
>>> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which
>>> are
>>> > close to completion.
>>> >
>>> > Folks who are working/reviewing these, kindly prioritize accordingly so
>>> > that we can make the release on time.
>>> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>>> >
>>> > Thanks in advance!
>>> >
>>> > -Subru/Arun
>>> >
>>> >
>>> >
>>>
>>
>>
>

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Lei Xu]

Hey,  Weiwei and Jitendra

Thanks a lot for this large effort to bring us ozone.

* As the current state of Ozone implementation, what are the major
benefits of using today’s Ozone over HDFS?  Giving that its missing
features like HDFS-12680 and HDFS-12697, being disabled by default,
and the closing of Hadoop 3.0 release, should we wait for a late merge
when Ozone is more mature ? Or more generally, why should this merge
to a release branch happen now, when Ozone is not yet usable by users?
Staying on a feature branch seems like it's still the right place to
me.
* For the existing HDFS user, could you address the semantic gaps
between Ozone / Ozone File System and HDFS. It would be great to
illustrate what is the expected use cases for Ozone giving its
different architecture and design decisions?  Like no append, no
atomic rename and etc.
* A follow question, was it able to run any of today’s Hadoop
applications (MR, Spark, Impala, Presto and etc) on Ozone directly, or
against OZoneFileSystem? I think a performance / scalability gain or
extended functionality should be the prerequisites for the merge.
Additionally, I believe such tests will reveal the potential caveats
if any.
* Ozone’s architecture shows great potential to address NN
scalability.  However it looks like a XXL effort to me, considering
the fact that 1) the community had multiple unfinished attempts to
simply separate namespace and block management within the same NN
process, and 2) many existing features like snapshot, append, erasure
coding, and etc, are not straightforward to be implemented in today’s
ozone design. Could you share your opinions on this matter?
* How stable is the ozone client? Should we mark them as unstable for
now? Also giving the significant difference between OzoneClient and
HdfsClient, should move it to a separated package or even a project? I
second Konstantin’s option to separate ozone from HDFS.
* Please add sections to the end-user and system admin oriented
documents for deploying and operating SCM, KSM, and also the chunk
servers on DataNodes. Additionally, the introduction in
“OZoneGettingStarted.md” is still building ozone from feature branch
HDFS-7240.

Best regards,

On Mon, Oct 23, 2017 at 11:10 AM, Jitendra Pandey
 wrote:
> I have filed https://issues.apache.org/jira/browse/HDFS-12697 to ensure ozone 
> stays disabled in a secure environment.
> Since ozone is disabled by default and will not come with security on, it 
> will not expose any new attack surface in a Hadoop deployment.
> Ozone security effort will need a detailed design and discussion on a 
> community jira. Hopefully, that effort will start soon after the merge.
>
> Thanks
> jitendra
>
> On 10/20/17, 2:40 PM, "larry mccay"  wrote:
>
> All -
>
> I broke this list of questions out into a separate DISCUSS thread where we
> can iterate over how a security audit process at merge time might look and
> whether it is even something that we want to take on.
>
> I will try and continue discussion on that thread and drive that to some
> conclusion before bringing it into any particular merge discussion.
>
> thanks,
>
> --larry
>
> On Fri, Oct 20, 2017 at 12:37 PM, larry mccay  wrote:
>
> > I previously sent this same email from my work email and it doesn't seem
> > to have gone through - resending from apache account (apologizing up 
> from
> > for the length)
> >
> > For such sizable merges in Hadoop, I would like to start doing security
> > audits in order to have an initial idea of the attack surface, the
> > protections available for known threats, what sort of configuration is
> > being used to launch processes, etc.
> >
> > I dug into the architecture documents while in the middle of this list -
> > nice docs!
> > I do intend to try and make a generic check list like this for such
> > security audits in the future so a lot of this is from that but I tried 
> to
> > also direct specific questions from those docs as well.
> >
> > 1. UIs
> > I see there are at least two UIs - Storage Container Manager and Key 
> Space
> > Manager. There are a number of typical vulnerabilities that we find in 
> UIs
> >
> > 1.1. What sort of validation is being done on any accepted user input?
> > (pointers to code would be appreciated)
> > 1.2. What explicit protections have been built in for (pointers to code
> > would be appreciated):
> >   1.2.1. cross site scripting
> >   1.2.2. cross site request forgery
> >   1.2.3. click jacking (X-Frame-Options)
> > 1.3. What sort of authentication is required for access to the UIs?
> > 1.4. What authorization is available for determining who can access what
> > capabilities of the UIs for either viewing, modifying data or affecting
> > object stores and related processes?
> > 1.5. Are the UIs built