[RESULT] [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Junping Du]

Thanks to all who verified and voted!

I give my binding +1 to conclude the vote for 2.8.3 RC0, based on:
- Build from source and verify signatures
- Deploy pseudo-distributed cluster and run some simple job, like: pi, sleep, 
etc.
- Verify UI of daemons, like: NameNode, DataNode, ResourceManager, NodeManager, 
etc

- Verify rolling upgrade features from 2.7.4, include MR over distributed 
cache, NM restart with work preserving, etc.

Now, we have:

8 binding +1s, from:
 Eric Payne, Jason Lowe, Jian He, Wangda Tan, Rohith Sharma K S, Sunil G, 
Naganarasimha Garla, Junping Du

5 non-binding +1s, from:
Brahma Reddy Battula, Kuhu Shukla, Ajay Kumar, Eric Badger, Chandni Singh

and no -1s.

So I am glad to announce that the vote of 2.8.3 RC0 passes.

Thanks everyone listed above who tried the release candidate and vote and all 
who ever help with 2.8.3 release effort in all kinds of ways.
I'll push the release bits and send out an announcement for 2.8.3 soon.

Thanks,

Junping


From: Naganarasimha Garla <naganarasimha...@apache.org>
Sent: Wednesday, December 13, 2017 9:39 AM
To: Sunil G
Cc: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

Thanks Junping for the release. +1 (binding)

Tested on a single node pseudo cluster, I performed the following tests:
- Downloaded the tars and verified the signatures, installed using the tar
- Successfully ran few MR jobs
- Verified few hdfs operations
- Verified RM,NM and HDFS webui's
- configured labels and submitted some apps

Thanks and Regards,
+ Naga

On Wed, Dec 13, 2017 at 8:14 PM, Sunil G 
<sun...@apache.org<mailto:sun...@apache.org>> wrote:
+1 (binding)

Thanks Junping for the effort.
I have deployed a cluster built from source tar ball.


   - Ran few MR apps and verified UI. CLI commands are also fine related to
   app.
   - Tested below feature sanity
  - Application priority
  - Application timeout
   - Tested basic NodeLabel scenarios.
  - Added some labels to couple of nodes
  - Verified old UI for labels
  - Submitted apps to labelled cluster and it works fine.
  - Also performed few cli commands related to nodelabel
   - Test basic HA cases


Thanks
Sunil G


On Tue, Dec 5, 2017 at 3:28 PM Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:

> Hi all,
>  I've created the first release candidate (RC0) for Apache Hadoop
> 2.8.3. This is our next maint release to follow up 2.8.2. It includes 79
> important fixes and improvements.
>
>   The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0
>
>   The RC tag in git is: release-2.8.3-RC0
>
>   The maven artifacts are available via 
> repository.apache.org<http://repository.apache.org> at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1072
>
>   Please try the release and vote; the vote will run for the usual 5
> working days, ending on 12/12/2017 PST time.
>
> Thanks,
>
> Junping
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Junping Du]

Thanks Andrew for pushing new RC for 3.0.0. I was out last week, just get 
chance to validate new RC now.

Basically, I found two critical issues with the same rolling upgrade scenario 
as where HADOOP-15059 get found previously:
HDFS-12920, we changed value format for some hdfs configurations that old 
version MR client doesn't understand when fetching these configurations. Some 
quick workarounds are to add old value (without time unit) in hdfs-site.xml to 
override new default values but will generate many annoying warnings. I 
provided my fix suggestions on the JIRA already for more discussion.
The other one is YARN-7646. After we workaround HDFS-12920, will hit the issue 
that old version MR AppMaster cannot communicate with new version of YARN RM - 
could be related to resource profile changes from YARN side but root cause are 
still in investigation.

The first issue may not belong to a blocker given we can workaround this 
without code change. I am not sure if we can workaround 2nd issue so far. If 
not, we may have to fix this or compromise with withdrawing support of rolling 
upgrade or calling it a stable release.


Thanks,

Junping


From: Robert Kanter 
Sent: Tuesday, December 12, 2017 3:10 PM
To: Arun Suresh
Cc: Andrew Wang; Lei Xu; Wei-Chiu Chuang; Ajay Kumar; Xiao Chen; Aaron T. 
Myers; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

+1 (binding)

+ Downloaded the binary release
+ Deployed on a 3 node cluster on CentOS 7.3
+ Ran some MR jobs, clicked around the UI, etc
+ Ran some CLI commands (yarn logs, etc)

Good job everyone on Hadoop 3!


- Robert

On Tue, Dec 12, 2017 at 1:56 PM, Arun Suresh  wrote:

> +1 (binding)
>
> - Verified signatures of the source tarball.
> - built from source - using the docker build environment.
> - set up a pseudo-distributed test cluster.
> - ran basic HDFS commands
> - ran some basic MR jobs
>
> Cheers
> -Arun
>
> On Tue, Dec 12, 2017 at 1:52 PM, Andrew Wang 
> wrote:
>
> > Hi everyone,
> >
> > As a reminder, this vote closes tomorrow at 12:31pm, so please give it a
> > whack if you have time. There are already enough binding +1s to pass this
> > vote, but it'd be great to get additional validation.
> >
> > Thanks to everyone who's voted thus far!
> >
> > Best,
> > Andrew
> >
> >
> >
> > On Tue, Dec 12, 2017 at 11:08 AM, Lei Xu  wrote:
> >
> > > +1 (binding)
> > >
> > > * Verified src tarball and bin tarball, verified md5 of each.
> > > * Build source with -Pdist,native
> > > * Started a pseudo cluster
> > > * Run ec -listPolicies / -getPolicy / -setPolicy on /  , and run hdfs
> > > dfs put/get/cat on "/" with XOR-2-1 policy.
> > >
> > > Thanks Andrew for this great effort!
> > >
> > > Best,
> > >
> > >
> > > On Tue, Dec 12, 2017 at 9:55 AM, Andrew Wang  >
> > > wrote:
> > > > Hi Wei-Chiu,
> > > >
> > > > The patchprocess directory is left over from the create-release
> > process,
> > > > and it looks empty to me. We should still file a create-release JIRA
> to
> > > fix
> > > > this, but I think this is not a blocker. Would you agree?
> > > >
> > > > Best,
> > > > Andrew
> > > >
> > > > On Tue, Dec 12, 2017 at 9:44 AM, Wei-Chiu Chuang <
> weic...@cloudera.com
> > >
> > > > wrote:
> > > >
> > > >> Hi Andrew, thanks the tremendous effort.
> > > >> I found an empty "patchprocess" directory in the source tarball,
> that
> > is
> > > >> not there if you clone from github. Any chance you might have some
> > > leftover
> > > >> trash when you made the tarball?
> > > >> Not wanting to nitpicking, but you might want to double check so we
> > > don't
> > > >> ship anything private to you in public :)
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Dec 12, 2017 at 7:48 AM, Ajay Kumar <
> > ajay.ku...@hortonworks.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> +1 (non-binding)
> > > >>> Thanks for driving this, Andrew Wang!!
> > > >>>
> > > >>> - downloaded the src tarball and verified md5 checksum
> > > >>> - built from source with jdk 1.8.0_111-b14
> > > >>> - brought up a pseudo distributed cluster
> > > >>> - did basic file system operations (mkdir, list, put, cat) and
> > > >>> confirmed that everything was working
> > > >>> - Run word count, pi and DFSIOTest
> > > >>> - run hdfs and yarn, confirmed that the NN, RM web UI worked
> > > >>>
> > > >>> Cheers,
> > > >>> Ajay
> > > >>>
> > > >>> On 12/11/17, 9:35 PM, "Xiao Chen"  wrote:
> > > >>>
> > > >>> +1 (binding)
> > > >>>
> > > >>> - downloaded src tarball, verified md5
> > > >>> - built from source with jdk1.8.0_112
> > > >>> - started a pseudo cluster with hdfs and kms
> > > >>> - sanity checked encryption related operations working
> > > >>> - sanity checked webui and logs.
> > > >>>
> > > >>>

[jira] [Created] (YARN-7646) MR job (based on old version tarball) get failed due to incompatible resource request

[Junping Du (JIRA)]

Junping Du created YARN-7646:


 Summary: MR job (based on old version tarball) get failed due to 
incompatible resource request
 Key: YARN-7646
 URL: https://issues.apache.org/jira/browse/YARN-7646
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn
Reporter: Junping Du
Priority: Blocker


With quick workaround with fixing HDFS-12920 (set non time unit to 
hdfs-site.xml), the job still get failed with following error:
{noformat}
2017-12-12 16:39:13,105 ERROR [RMCommunicator Allocator] 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM. 
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid 
resource request, requested memory < 0, or requested memory > max configured, 
requestedMemory=-1, maxMemory=8192
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:275)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:240)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:256)
at 
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:246)
at 
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:217)
at 
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at 
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:388)
at 
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at 
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at 
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at 
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateYarnException(RPCUtil.java:75)
at 
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:116)
at 
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy81.allocate(Unknown Source)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:206)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:783)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:280)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:279)
at java.lang.Thread.run(Thread.java:745)
{noformat}
It looks like incompatible change with communication between old MR 

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Junping Du]

Hi Konstantin,

 Thanks for verification and comments. I was verifying your example below 
but found it is actually matched:


jduMBP:hadoop-2.8.3 jdu$ md5 ~/Downloads/hadoop-2.8.3-src.tar.gz
MD5 (/Users/jdu/Downloads/hadoop-2.8.3-src.tar.gz) = 
e53d04477b85e8b58ac0a26468f04736

What's your md5 checksum for given source tar ball?


Thanks,


Junping



From: Konstantin Shvachko <shv.had...@gmail.com>
Sent: Saturday, December 9, 2017 11:06 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

Hey Junping,

Could you pls upload mds relative to the tar.gz etc. files rather than their 
full path
/build/source/target/artifacts/hadoop-2.8.3-src.tar.gz:
   MD5 = E5 3D 04 47 7B 85 E8 B5  8A C0 A2 64 68 F0 47 36

Otherwise mds don't match for me.

Thanks,
--Konstantin

On Tue, Dec 5, 2017 at 1:58 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi all,
 I've created the first release candidate (RC0) for Apache Hadoop 2.8.3. 
This is our next maint release to follow up 2.8.2. It includes 79 important 
fixes and improvements.

  The RC artifacts are available at: 
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0

  The RC tag in git is: release-2.8.3-RC0

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1072

  Please try the release and vote; the vote will run for the usual 5 
working days, ending on 12/12/2017 PST time.

Thanks,

Junping

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Junping Du]

Thanks Eric for verification!

A kindly reminder: The original due date for 2.8.3 voting is tomorrow but I 
only receive 1 binding vote so far - we have 76 PMCs and 127 Committers!

I can understand the whole community are busy with 3 release RC voting (2.7.5, 
2.8.3 and 3.0.0) and may be it is necessary to extend the voting period for a 
few more days. But please try as much as possible to verify our release bits. 
Thanks!


Thanks,


Junping?



From: Eric Payne <eric.payne1...@yahoo.com>
Sent: Monday, December 11, 2017 1:51 PM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

Thanks Junping for the hard work on this release.

+1 (binding)

On a 6 node pseudo cluster (4 NMs), I performed the following manual tests:

- Built and installed from source

- Successfully ran a stream job

- Verified that user weights are honored by assigning the appropriate amount of 
resources to the weighted users.

- Ensured that FiarOrderingPolicy and FifoOrderingPolicy worked in the Capacity 
Scheduler as expected

- Applications with higher priorities are assigned containers as expected in 
the FifoOrderingPolicy of the Capacity Scheduler until the user reaches its 
user resource limit.

Eric Payne



From: Junping Du <j...@hortonworks.com>
To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>; 
"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
Sent: Tuesday, December 5, 2017 3:58 AM
Subject: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

Hi all,
I've created the first release candidate (RC0) for Apache Hadoop 2.8.3. 
This is our next maint release to follow up 2.8.2. It includes 79 important 
fixes and improvements.

  The RC artifacts are available at: 
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0

  The RC tag in git is: release-2.8.3-RC0

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1072

  Please try the release and vote; the vote will run for the usual 5 
working days, ending on 12/12/2017 PST time.

Thanks,

Junping

[VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Junping Du]

Hi all,
 I've created the first release candidate (RC0) for Apache Hadoop 2.8.3. 
This is our next maint release to follow up 2.8.2. It includes 79 important 
fixes and improvements.

  The RC artifacts are available at: 
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0

  The RC tag in git is: release-2.8.3-RC0

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1072

  Please try the release and vote; the vote will run for the usual 5 
working days, ending on 12/12/2017 PST time.

Thanks,

Junping

Re: Apache Hadoop 2.8.3 Release Plan

[Junping Du]

Thanks Konstantin and Weiwei for calling attention. These two HDFS issues looks 
to be important to be fixed which are on my radar now.

I will hold on RC cut until we figure them out.


Thanks,


Junping



From: Weiwei Yang <cheersy...@hotmail.com>
Sent: Tuesday, November 21, 2017 7:45 PM
To: Junping Du; Konstantin Shvachko
Cc: Wangda Tan; Andrew Wang; Zheng, Kai; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

Agree with Konstantin. This two issues has been opened for a while but could 
not reach a consensus on the fix, hope this gets enough attention from the 
community and get them resolved.

Thanks

--
Weiwei

On 22 Nov 2017, 11:18 AM +0800, Konstantin Shvachko <shv.had...@gmail.com>, 
wrote:
I would consider these two blockers for 2.8.3 as they crash NN:

https://issues.apache.org/jira/browse/HDFS-12638
https://issues.apache.org/jira/browse/HDFS-12832

Thanks,
--Konstantin

On Tue, Nov 21, 2017 at 11:16 AM, Junping Du <j...@hortonworks.com> wrote:

Thanks Andrew and Wangda for comments!

To me, an improvement with 17 patches is not a big problem as this is
self-contained and I didn't see a single line of delete/update on existing
code - well, arguably, patches with only adding code can also have big
impact but not likely the case here.

While the dependency discussions on HADOOP-14964 are still going on, I
will leave the decision to JIRA discussion based on which approach we will
choose(shaded?) and impact. If we cannot make consensus in short term,
probably we have to miss this in 2.8.3 release.


Okay. Last call for blocker/critical fixes landing on branch-2.8.3. RC0
will get cut-off shortly.



Thanks,


Junping



From: Wangda Tan <wheele...@gmail.com
Sent: Tuesday, November 21, 2017 10:52 AM
To: Andrew Wang
Cc: Junping Du; Zheng, Kai; common-...@hadoop.apache.org;
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

Thanks Junping for driving this.

For the bug fix vs. improvement, it is actually very hard to define,
improvement could be self-contained and useful, bug fix could be dangerous
in some cases. To me, If an improvement fixed some existing use case, and
the fix is self-contained. I will be open to bring such fix to maintenance
release. For example, in 2.8.2, we back ported CapacityScheduler intra
queue preemption https://issues.apache.org/jira/browse/YARN-2113. It is a
big change in terms of patch size, but since it fixes broken use case
(balance user usage under Capacity Scheduler leaf queue), we backported it
to 2.8.2 after thorough tests and validations by Yahoo.

I'm not quite familiar with HADOOP-14964, I will leave the decision to
committers who know more about the field.

Just my 2 cents.

Regards,
Wangda


On Tue, Nov 21, 2017 at 10:21 AM, Andrew Wang <andrew.w...@cloudera.com<
mailto:andrew.w...@cloudera.com>> wrote:
The Aliyun OSS code isn't a small improvement. If you look at Sammi's
comment
<https://issues.apache.org/jira/browse/HADOOP-14964?
focusedCommentId=16247085=com.atlassian.jira.
plugin.system.issuetabpanels:comment-tabpanel#comment-16247085>,
it's
a 17 patch series that is being backported in one shot. What we're talking
about is equivalent to merging a feature branch in a maintenance release. I
see that Kai and Chris are having a discussion about the dependency
changes, which indicates this is not a zero-risk change either. We really
should not be changing dependency versions in a maintenance unless it's
because of a bug.

It's unfortunate from a timing perspective that this missed 2.9.0, but I
still think it should wait for the next minor. Merging a feature into a
maintenance release sets the wrong precedent.

Best,
Andrew

On Tue, Nov 21, 2017 at 1:08 AM, Junping Du <j...@hortonworks.com<mailto:jd
u...@hortonworks.com>> wrote:

Thanks Kai for calling out this feature/improvement for attention and
Andrew for comments.


While I agree that maintenance release should focus on important bug fix
only, I doubt we have strict rules to disallow any features/improvements
to
land on maint release especially when those are small footprint or low
impact on existing code/features. In practice, we indeed has 77 new
features/improvements in latest 2.7.3 and 2.7.4 release.


Back to HADOOP-14964, I did a quick check and it looks like case here
belongs to self-contained improvement that has very low impact on
existing
code base, so I am OK with the improvement get landed on branch-2.8 in
case
it is well reviewed and tested.


However, as RM of branch-2.8, I have two concerns to accept it in our
2.8.3 release:

1. Timing - as I mentioned in beginning, the main purpose of 2.8.3 are
for
several critical bug fixes and we should target to release it very soon -
my cu

Re: [DISCUSS] A final minor release off branch-2?

[Junping Du]

Hi Andrew,

bq. Source and binary compatibility are not required for 3.0.0. It's a new 
major release, and there are known, documented incompatibilities in this regard.

Technically, it is true. However, in practically, we should retain 
compatibility as much as we can. Otherwise, we could break downstream projects, 
third-party libraries and existing users applications unintentionally. A quick 
example here is a blocker issue I just reported in HADOOP-15059 which break old 
(2.x) MR application with 3.0 deployment - due to token format incompatible 
issue.


bq. To follow up on my earlier email, I don't think there's need for a bridge 
release given that we've successfully tested rolling upgrade from 2.x to 3.0.0.​

Did we find the same issue as HADOOP-15059? If so, just curious on what rolling 
upgrade means here - IMO, upgrade with breaking running applications shouldn't 
be recognized as "rolling". Do I miss anything?



Thanks,


Junping



From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Wednesday, November 15, 2017 10:34 AM
To: Junping Du
Cc: Wangda Tan; Steve Loughran; Vinod Kumar Vavilapalli; Kai Zheng; Arun 
Suresh; common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: [DISCUSS] A final minor release off branch-2?

Hi Junping,

On Wed, Nov 15, 2017 at 1:37 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Thanks Vinod to bring up this discussion, which is just in time.

I agree with most responses that option C is not a good choice as our community 
bandwidth is precious and we should focus on very limited mainstream branches 
to develop, test and deployment. Of course, we should still follow Apache way 
to allow any interested committer for rolling up his/her own release given 
specific requirement over the mainstream releases.

I am not biased on option A or B (I will discuss this later), but I think a 
bridge release for upgrading to and back from 3.x is very necessary.
The reasons are obviously:
1. Given lesson learned from previous experience of migration from 1.x to 2.x, 
no matter how careful we tend to be, there is still chance that some level of 
compatibility (source, binary, configuration, etc.) get broken for the 
migration to new major release. Some of these incompatibilities can only be 
identified in runtime after GA release with widely deployed in production 
cluster - we have tons of downstream projects and numerous configurations and 
we cannot cover them all from in-house deployment and test.

Source and binary compatibility are not required for 3.0.0. It's a new major 
release, and there are known, documented incompatibilities in this regard.

That said, we've done far, far more in this regard compared to previous major 
or minor releases. We've compiled all of CDH against Hadoop 3 and run our suite 
of system tests for the platform. We've been testing in this way since 
3.0.0-alpha1 and found and fixed plenty of source and binary compatibility 
issues during the alpha and beta process. Many of these fixes trickled down 
into 2.8 and 2.9.

2. From recent classpath isolation work, I was surprised to find out that many 
of our downstream projects (HBase, Tez, etc.) are still consuming many 
non-public, server side APIs of Hadoop, not saying the projects/products 
outside of hadoop ecosystem. Our API compatibility test does not (and should 
not) cover these cases and situations. We can claim that new major release 
shouldn't be responsible for these private API changes. But given the 
possibility of breaking existing applications in some way, users could be very 
hesitated to migrate to 3.x release if there is no safe solution to roll back.

This is true for 2.x releases as well. Similar to the previous answer, we've 
compiled all of CDH against Hadoop 3, providing a much higher level of 
assurance even compared to 2.x releases.

3. Beside incompatibilities, there is also possible to have performance 
regressions (lower throughput, higher latency, slower job running, bigger 
memory footprint or even memory leaking, etc.) for new hadoop releases. While 
the performance impact of migration (if any) could be neglectable to some 
users, other users could be very sensitive and wish to roll back if it happens 
on their production cluster.

Yes, bugs exist. I won't claim that 3.0.0 is bug-free. All new releases can 
potentially introduce new bugs.

However, I don't think rollback is the solution. In my experience, users rarely 
rollback since it's so disruptive and causes data loss. It's much more common 
that they patch and upgrade. With that in mind, I'd rather we spend our effort 
on making 3.0.x high-quality vs. making it easier to rollback.

The root of my concern in announcing a "bridge release" is that it discourages 
users from upgrading to 3.0.0 until a bridge release is out. I strongly believe 
the level of quality provid

Re:

[Junping Du]

Just filed HADOOP-15059 to track this.


Thanks,


Junping



From: Junping Du
Sent: Tuesday, November 21, 2017 1:09 PM
To: Vinod Kumar Vavilapalli; Allen Wittenauer
Cc: common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

Another issue I just found is we have token format compatibility issue between 
2.x and 3.0. I tried to run a simple MR job on 3.0 RC0 against with 2.9.0 
tarball which is failed. This incompatibility change should also break other 
applications which will break rolling upgrade.
I think it is another blocker for 3.0.0 release.

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Tuesday, November 21, 2017 12:59 PM
To: Allen Wittenauer
Cc: common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

>>> - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>>
>>
>> The yarn-ui profile has always been off by default, AFAIK. It's documented
>> to turn it on in BUILDING.txt for release builds, and we do it in
>> create-release.
>>
>> IMO not a blocker. I think it's also more of a dev question (do we want to
>> do this on every YARN build?) than a release one.
>
>   -1 on making yarn-ui always build.
>
>   For what is effectively an optional component (the old UI is still 
> there), it's heavy dependency requirements make it a special burden outside 
> of the Docker container.


Thanks for the background on this.

I got schooled offline too about the heaviness of the dependencies.


> If it can be changed such that it either always downloads the necessary bits 
> (regardless of the OS/chipset!) and/or doesn't kill the maven build if those 
> bits can't be found  (i.e., truly optional), then I'd be less opposed.  (and, 
> actually, quite pleased because then the docker image build would be 
> significantly faster.)


This is a good idea (to the extend I understand your proposal). Long term, we'd 
like YARN UI2 to replace current UI, so it shouldn't be optionally build and 
hence the build should be fast. Let me track this separately as a non-blocker.

Thanks
+Vinod
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Junping Du]

Another issue I just found is we have token format compatibility issue between 
2.x and 3.0. I tried to run a simple MR job on 3.0 RC0 against with 2.9.0 
tarball which is failed. This incompatibility change should also break other 
applications which will break rolling upgrade.
I think it is another blocker for 3.0.0 release.

Thanks,

Junping

From: Vinod Kumar Vavilapalli 
Sent: Tuesday, November 21, 2017 12:59 PM
To: Allen Wittenauer
Cc: common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

>>> - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>>
>>
>> The yarn-ui profile has always been off by default, AFAIK. It's documented
>> to turn it on in BUILDING.txt for release builds, and we do it in
>> create-release.
>>
>> IMO not a blocker. I think it's also more of a dev question (do we want to
>> do this on every YARN build?) than a release one.
>
>   -1 on making yarn-ui always build.
>
>   For what is effectively an optional component (the old UI is still 
> there), it’s heavy dependency requirements make it a special burden outside 
> of the Docker container.


Thanks for the background on this.

I got schooled offline too about the heaviness of the dependencies.


> If it can be changed such that it either always downloads the necessary bits 
> (regardless of the OS/chipset!) and/or doesn’t kill the maven build if those 
> bits can’t be found  (i.e., truly optional), then I’d be less opposed.  (and, 
> actually, quite pleased because then the docker image build would be 
> significantly faster.)


This is a good idea (to the extend I understand your proposal). Long term, we'd 
like YARN UI2 to replace current UI, so it shouldn't be optionally build and 
hence the build should be fast. Let me track this separately as a non-blocker.

Thanks
+Vinod
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.3 Release Plan

[Junping Du]

Thanks Andrew and Wangda for comments!

To me, an improvement with 17 patches is not a big problem as this is 
self-contained and I didn't see a single line of delete/update on existing code 
- well, arguably, patches with only adding code can also have big impact but 
not likely the case here.

While the dependency discussions on HADOOP-14964 are still going on, I will 
leave the decision to JIRA discussion based on which approach we will 
choose(shaded?) and impact. If we cannot make consensus in short term, probably 
we have to miss this in 2.8.3 release.


Okay. Last call for blocker/critical fixes landing on branch-2.8.3. RC0 will 
get cut-off shortly.



Thanks,


Junping



From: Wangda Tan <wheele...@gmail.com>
Sent: Tuesday, November 21, 2017 10:52 AM
To: Andrew Wang
Cc: Junping Du; Zheng, Kai; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

Thanks Junping for driving this.

For the bug fix vs. improvement, it is actually very hard to define, 
improvement could be self-contained and useful, bug fix could be dangerous in 
some cases. To me, If an improvement fixed some existing use case, and the fix 
is self-contained. I will be open to bring such fix to maintenance release. For 
example, in 2.8.2, we back ported CapacityScheduler intra queue preemption 
https://issues.apache.org/jira/browse/YARN-2113. It is a big change in terms of 
patch size, but since it fixes broken use case (balance user usage under 
Capacity Scheduler leaf queue), we backported it to 2.8.2 after thorough tests 
and validations by Yahoo.

I'm not quite familiar with HADOOP-14964, I will leave the decision to 
committers who know more about the field.

Just my 2 cents.

Regards,
Wangda


On Tue, Nov 21, 2017 at 10:21 AM, Andrew Wang 
<andrew.w...@cloudera.com<mailto:andrew.w...@cloudera.com>> wrote:
The Aliyun OSS code isn't a small improvement. If you look at Sammi's
comment
<https://issues.apache.org/jira/browse/HADOOP-14964?focusedCommentId=16247085=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16247085>,
it's
a 17 patch series that is being backported in one shot. What we're talking
about is equivalent to merging a feature branch in a maintenance release. I
see that Kai and Chris are having a discussion about the dependency
changes, which indicates this is not a zero-risk change either. We really
should not be changing dependency versions in a maintenance unless it's
because of a bug.

It's unfortunate from a timing perspective that this missed 2.9.0, but I
still think it should wait for the next minor. Merging a feature into a
maintenance release sets the wrong precedent.

Best,
Andrew

On Tue, Nov 21, 2017 at 1:08 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:

> Thanks Kai for calling out this feature/improvement for attention and
> Andrew for comments.
>
>
> While I agree that maintenance release should focus on important bug fix
> only, I doubt we have strict rules to disallow any features/improvements to
> land on maint release especially when those are small footprint or low
> impact on existing code/features. In practice, we indeed has 77 new
> features/improvements in latest 2.7.3 and 2.7.4 release.
>
>
> Back to HADOOP-14964, I did a quick check and it looks like case here
> belongs to self-contained improvement that has very low impact on existing
> code base, so I am OK with the improvement get landed on branch-2.8 in case
> it is well reviewed and tested.
>
>
> However, as RM of branch-2.8, I have two concerns to accept it in our
> 2.8.3 release:
>
> 1. Timing - as I mentioned in beginning, the main purpose of 2.8.3 are for
> several critical bug fixes and we should target to release it very soon -
> my current plan is to cut RC out within this week inline with waiting
> for 3.0.0 vote closing. Can this improvement be well tested against
> branch-2.8.3 within this strictly timeline? It seems a bit rush unless we
> have strong commitment on test plan and activities in such a tight time.
>
>
> 2. Upgrading - I haven't heard we settle down the plan of releasing this
> feature in 2.9.1 release - though I saw some discussions are going on
> at HADOOP-14964. Assume 2.8.3 is released ahead of 2.9.1 and it includes
> this improvement, then users consuming this feature/improvement have no 2.9
> release to upgrade or forcefully upgrade with regression. We may need a
> better upgrade story here.
>
>
> Pls let me know what you think. Thanks!
>
>
>
> Thanks,
>
>
> Junping
>
>
> ----------
> *From:* Andrew Wang 
> <andrew.w...@cloudera.com<mailto:andrew.w...@cloudera.com>>
> *Sent:* Monday, November 20, 20

Re: Apache Hadoop 2.8.3 Release Plan

[Junping Du]

Thanks Kai for calling out this feature/improvement for attention and Andrew 
for comments.


While I agree that maintenance release should focus on important bug fix only, 
I doubt we have strict rules to disallow any features/improvements to land on 
maint release especially when those are small footprint or low impact on 
existing code/features. In practice, we indeed has 77 new features/improvements 
in latest 2.7.3 and 2.7.4 release.


Back to HADOOP-14964, I did a quick check and it looks like case here belongs 
to self-contained improvement that has very low impact on existing code base, 
so I am OK with the improvement get landed on branch-2.8 in case it is well 
reviewed and tested.


However, as RM of branch-2.8, I have two concerns to accept it in our 2.8.3 
release:

1. Timing - as I mentioned in beginning, the main purpose of 2.8.3 are for 
several critical bug fixes and we should target to release it very soon - my 
current plan is to cut RC out within this week inline with waiting for 3.0.0 
vote closing. Can this improvement be well tested against branch-2.8.3 within 
this strictly timeline? It seems a bit rush unless we have strong commitment on 
test plan and activities in such a tight time.


2. Upgrading - I haven't heard we settle down the plan of releasing this 
feature in 2.9.1 release - though I saw some discussions are going on at 
HADOOP-14964. Assume 2.8.3 is released ahead of 2.9.1 and it includes this 
improvement, then users consuming this feature/improvement have no 2.9 release 
to upgrade or forcefully upgrade with regression. We may need a better upgrade 
story here.


Pls let me know what you think. Thanks!



Thanks,


Junping



From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Monday, November 20, 2017 10:22 PM
To: Zheng, Kai
Cc: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

I'm against including new features in maintenance releases, since they're meant 
to be bug-fix only.

If we're struggling with being able to deliver new features in a safe and 
timely fashion, let's try to address that, not overload the meaning of 
"maintenance release".

Best,
Andrew

On Mon, Nov 20, 2017 at 5:20 PM, Zheng, Kai 
<kai.zh...@intel.com<mailto:kai.zh...@intel.com>> wrote:
Hi Junping,

Thank you for making 2.8.2 happen and now planning the 2.8.3 release.

I have an ask, is it convenient to include the back port work for OSS connector 
module? We have some Hadoop users that wish to have it by default for 
convenience, though in the past they used it by back porting themselves. I have 
raised this and got thoughts from Chris and Steve. Looks like this is more 
wanted for 2.9 but I wanted to ask again here for broad feedback and thoughts 
by this chance. The back port patch is available for 2.8 and the one for 
branch-2 was already in. IMO, 2.8.x is promising as we can see some shift from 
2.7.x, hence it's worth more important features and efforts. How would you 
think? Thanks!

https://issues.apache.org/jira/browse/HADOOP-14964

Regards,
Kai

-Original Message-
From: Junping Du [mailto:j...@hortonworks.com<mailto:j...@hortonworks.com>]
Sent: Tuesday, November 14, 2017 9:02 AM
To: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>; 
yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>
Subject: Apache Hadoop 2.8.3 Release Plan

Hi,
We have several important fixes get landed on branch-2.8 and I would like 
to cut off branch-2.8.3 now to start 2.8.3 release work.
So far, I don't see any pending blockers on 2.8.3, so my current plan is to 
cut off 1st RC of 2.8.3 in next several days:
 -  For all coming commits to land on branch-2.8, please mark the fix 
version as 2.8.4.
 -  If there is a really important fix for 2.8.3 and getting closed, 
please notify me ahead before landing it on branch-2.8.3.
Please let me know if you have any thoughts or comments on the plan.

Thanks,

Junping

From: dujunp...@gmail.com<mailto:dujunp...@gmail.com> 
<dujunp...@gmail.com<mailto:dujunp...@gmail.com>> on behalf of ??? 
<junping...@apache.org<mailto:junping...@apache.org>>
Sent: Friday, October 27, 2017 3:33 PM
To: gene...@hadoop.apache.org<mailto:gene...@hadoop.apache.org>
Subject: [ANNOUNCE] Apache Hadoop 2.8.2 Release.

Hi all,

It gives me great pleasure to announce that the Apache Hadoop community has 
voted to release Apache Hadoop 2.8.2, which is now available for download from 
Apache mirrors[1]. For download instructions please refer to the Apache Hadoop 
Release page [2].

Apache Hadoop 2.8.2 is the firs

Re: [DISCUSS] A final minor release off branch-2?

[Junping Du]

Thanks Vinod to bring up this discussion, which is just in time.

I agree with most responses that option C is not a good choice as our community 
bandwidth is precious and we should focus on very limited mainstream branches 
to develop, test and deployment. Of course, we should still follow Apache way 
to allow any interested committer for rolling up his/her own release given 
specific requirement over the mainstream releases.

I am not biased on option A or B (I will discuss this later), but I think a 
bridge release for upgrading to and back from 3.x is very necessary. 
The reasons are obviously:
1. Given lesson learned from previous experience of migration from 1.x to 2.x, 
no matter how careful we tend to be, there is still chance that some level of 
compatibility (source, binary, configuration, etc.) get broken for the 
migration to new major release. Some of these incompatibilities can only be 
identified in runtime after GA release with widely deployed in production 
cluster - we have tons of downstream projects and numerous configurations and 
we cannot cover them all from in-house deployment and test.

2. From recent classpath isolation work, I was surprised to find out that many 
of our downstream projects (HBase, Tez, etc.) are still consuming many 
non-public, server side APIs of Hadoop, not saying the projects/products 
outside of hadoop ecosystem. Our API compatibility test does not (and should 
not) cover these cases and situations. We can claim that new major release 
shouldn't be responsible for these private API changes. But given the 
possibility of breaking existing applications in some way, users could be very 
hesitated to migrate to 3.x release if there is no safe solution to roll back. 

3. Beside incompatibilities, there is also possible to have performance 
regressions (lower throughput, higher latency, slower job running, bigger 
memory footprint or even memory leaking, etc.) for new hadoop releases. While 
the performance impact of migration (if any) could be neglectable to some 
users, other users could be very sensitive and wish to roll back if it happens 
on their production cluster.

As Andrew mentioned in early email threads, some work has been done for 
verifying rolling upgrade from 2.x to 3.0 (just curious that which 2.x release 
is tested to upgrade from? 2.8.2 or 2.9.0 which is still in releasing?). But I 
am not aware any work we are doing now to test downgrade from 3.0 to 2.x 
(correct me if I miss any work). If users hit any of three situations I 
mentioned above then we should give them the chance to roll back if they are 
really conservative to these unexpected side-effect of upgrading. Given this, 
we should have this bridge release to cover the case for 3.0 safely roll back 
(no matter rolling or not). I am not sure it should be 2.9.x or 2.10.x for now 
(we can just call it 2.BR release) because we are not sure what exactly changes 
we should include for supporting roll back from 3.0 at this moment. We can 
defer this decision to discuss later when we have better ideas.

Summary for my two cents:
- No more feature release should happen on branch-2. 2.9 or 2.10 should be the 
last minor release (mainstream of community) on branch-2

- A bridge release is necessary for safely upgrade/downgrade to 3.x

- We can decide later to see if 2.10 is necessary when scope of the bridge 
release is more clear.


Thanks,

Junping


From: Andrew Wang 
Sent: Tuesday, November 14, 2017 2:25 PM
To: Wangda Tan
Cc: Steve Loughran; Vinod Kumar Vavilapalli; Kai Zheng; Arun Suresh; 
common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: [DISCUSS] A final minor release off branch-2?

To follow up on my earlier email, I don't think there's need for a bridge
release given that we've successfully tested rolling upgrade from 2.x to
3.0.0. I expect we'll keep making improvements to smooth over any
additional incompatibilities found, but there isn't a requirement that a
user upgrade to a bridge release before upgrading to 3.0.

Otherwise, I don't have a strong opinion about when to discontinue branch-2
releases. Historically, a release line is maintained until interest in it
wanes. If the maintainers are taking care of the backports, it's not much
work for the rest of us to vote on the RCs.

Best,
Andrew

On Mon, Nov 13, 2017 at 4:19 PM, Wangda Tan  wrote:

> Thanks Vinod for staring this,
>
> I'm also leaning towards the plan (A):
>
>
>
>
> * (A)-- Make 2.9.x the last minor release off branch-2-- Have a
> maintenance release that bridges 2.9 to 3.x-- Continue to make more
> maintenance releases on 2.8 and 2.9 as necessary*
>
> The only part I'm not sure is having a separate bridge release other than
> 3.x.
>
> For the bridge release, Steve's suggestion sounds more doable:
>
> ** 3.1+ for new features*
> ** fixes to 3.0.x &, where 

Apache Hadoop 2.8.3 Release Plan

[Junping Du]

Hi,
We have several important fixes get landed on branch-2.8 and I would like 
to cut off branch-2.8.3 now to start 2.8.3 release work. 
So far, I don't see any pending blockers on 2.8.3, so my current plan is to 
cut off 1st RC of 2.8.3 in next several days: 
 -  For all coming commits to land on branch-2.8, please mark the fix 
version as 2.8.4.
 -  If there is a really important fix for 2.8.3 and getting closed, 
please notify me ahead before landing it on branch-2.8.3.
Please let me know if you have any thoughts or comments on the plan.

Thanks,

Junping

From: dujunp...@gmail.com  on behalf of 俊平堵 

Sent: Friday, October 27, 2017 3:33 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.2 Release.

Hi all,

It gives me great pleasure to announce that the Apache Hadoop community
has
voted to release Apache Hadoop 2.8.2, which is now available for download
from
Apache mirrors[1]. For download instructions please refer to the Apache
Hadoop
Release page [2].

Apache Hadoop 2.8.2 is the first GA release of Apache Hadoop 2.8 line and
our
newest stable release for entire Apache Hadoop project. For major changes
incuded in Hadoop 2.8 line, please refer Hadoop 2.8.2 main page[3].

This release has 315 resolved issues since previous 2.8.1 release with
following
breakdown:
   - 91 in Hadoop Common
   - 99 in HDFS
   - 105 in YARN
   - 20 in MapReduce
Please read the log of CHANGES[4] and RELEASENOTES[5] for more details.

The release news is posted on the Hadoop website too, you can go to the
downloads section directly [6].

Thank you all for contributing to the Apache Hadoop release!


Cheers,

Junping


[1] http://www.apache.org/dyn/closer.cgi/hadoop/common

[2] http://hadoop.apache.org/releases.html

[3] http://hadoop.apache.org/docs/r2.8.2/index.html

[4]
http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/release/2.8.2/CHANGES.2.8.2.html

[5]
http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/release/2.8.2/RELEASENOTES.2.8.2.html

[6] http://hadoop.apache.org/releases.html#Download


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Cutting branch-2.9

[Junping Du]

Below is some findings from my recently run of JACC 
(https://builds.apache.org/job/Hadoop-2.9-JACC/12/artifact/target/compat-check/report.html)
 job against 2.9 and 2.8.2. I have discussed with Arun and Subru offline on 
jdiff report who convinced me some of items are not a big concern. Just tried 
to list here in case people in community have different voices:

 My original concerns of incompatibilities:
 1. It looks like we change from log4j to sel4j (HADOOP-12956) in 2.9 which 
is a huge incompatible change. Do we have consensus on this in community - as I 
saw HBase community are deny this kind of change in minor release? I talked 
with several other guys who seems to have different opinion.

2. Some APIs (deprecated in 2.8) get removed. My understanding is we only 
remove deprecated API in next major release (3.0) but not minor release. - Arun 
and Subru convinced me that removed 2 methods are not supposed to be used by 
other downstream projects

3. Some stable APIs get removed, like: YarnScheduler.allocate(), etc. We 
may need to add it back for compatibility of Scheduler API.

Thoughts?

Thanks,

Junping


From: Arun Suresh 
Sent: Thursday, November 2, 2017 11:55 PM
To: 俊平堵
Cc: Arun Suresh; su...@apache.org; common-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org
Subject: Re: Cutting branch-2.9

Quick update folks.

We just cut branch-2.9.0, and set the mvn version to 2.9.0
We've also updated the version on branch-2.9 to 2.9.1-SNAPSHOT.

Expect an RC in the next 24 hrs :)

Cheers
-Arun/Subru

On Tue, Oct 31, 2017 at 5:24 PM, Arun Suresh  wrote:

> Hi Junping,
>
> > In the mean time, branch-2.9 should be reserved for next 2.9 point
> release (2.9.1) and branch-2 should be reserved for next minor release
> (2.10.0 or whatever name it is). Thoughts?
> Yup, That is the current plan. We've updated mvn versions in branch-2 to
> point to 2.10.0-SNAPSHOT.
> Once we cut branch-2.9.0, we will update mvn versions on branch-2.9.0 to
> 2.9.0 and set versions on branch-2.9 to 2.9.1-SNAPSHOT (We plan to do that
> by Thursday)
>
> Cheers
> -Arun
>
> On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵  wrote:
>
>> Hi Arun/Subru,
>> Thanks for updating on 2.9.0 release progress. A quick question here:
>> are we planning to release from branch-2.9 directly?
>> I doubt this as it seems against our current branch release practice (
>> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of
>> any confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release
>> work. In the mean time, branch-2.9 should be reserved for next 2.9 point
>> release (2.9.1) and branch-2 should be reserved for next minor release
>> (2.10.0 or whatever name it is). Thoughts?
>>
>> bq. @Junping, lets move the jdiff conversation to separate thread.
>> Sure. I will reply jdiff in separated thread.
>>
>> Thanks,
>>
>> Junping
>>
>> 2017-10-31 13:44 GMT-07:00 Arun Suresh :
>>
>>> Hello folks
>>>
>>> We just cut branch-2.9 since all the critical/blocker issues are now
>>> resolved.
>>> We plan to perform some sanity checks for the rest of the week and cut
>>> branch-2.9.0 and push out an RC0 by the end of the week.
>>>
>>> Kindly refrain from committing to branch-2.9 without giving us a heads
>>> up.
>>>
>>> @Junping, lets move the jdiff conversation to separate thread.
>>>
>>> Cheers
>>> -Arun/Subru
>>>
>>> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan 
>>> wrote:
>>>
>>> > We want to give heads up that we are going to cut branch-2.9 tomorrow
>>> > morning.
>>> >
>>> > We are down to 3 blockers and they all are close to being committed
>>> > (thanks everyone):
>>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>>> >
>>> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which
>>> are
>>> > close to completion.
>>> >
>>> > Folks who are working/reviewing these, kindly prioritize accordingly so
>>> > that we can make the release on time.
>>> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>>> >
>>> > Thanks in advance!
>>> >
>>> > -Subru/Arun
>>> >
>>> >
>>> >
>>>
>>
>>
>


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Cutting branch-2.9

[Junping Du]

Hi Subru and Arun,
 Thanks for moving forward with 2.9 release. Is the first cut of 2.9 
release supposed to be a stable version or just an alpha version? If it is 
supposed to be a stable version, we should run jdiff test and check for API 
compatibility before releasing out. Please let me know if you need any help 
here.

Thanks,

Junping

From: Subru Krishnan 
Sent: Monday, October 30, 2017 12:39 PM
To: common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Cc: Arun Suresh
Subject: Cutting branch-2.9

We want to give heads up that we are going to cut branch-2.9 tomorrow
morning.

We are down to 3 blockers and they all are close to being committed (thanks
everyone):
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release

There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are
close to completion.

Folks who are working/reviewing these, kindly prioritize accordingly so
that we can make the release on time.
https://issues.apache.org/jira/browse/YARN-7398?filter=12342468

Thanks in advance!

-Subru/Arun



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[RESULT] [VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Junping Du]

??Thanks again for all who verified and voted!

I give my binding +1 to conclude the vote for 2.8.2 RC1, based on:
- Build from source and verify signatures
- Deploy pseudo-distributed cluster and run some simple job, like: pi, sleep, 
etc.
- Verify UI of daemons, like: NameNode, ResourceManager, NodeManager, etc.

Now, we have:

7 binding +1s, from:
 John Zhuge, Jason Lowe, Chris Douglas, Wangda Tan, Ravi Prakash, Eric 
Payne, Junping Du

10 non-binding +1s, from:
Hanisha Koneru, Wei Yan, Brahma Reddy Battula, Shane Kumpf, Ajay Kumar, 
Bharat Viswanadham, Mukul Kumar Singh, Eric Badger, Bibinchundatt, Rakesh 
Radhakrishnan

and no -1s.

So I am glad to announce that the vote of 2.8.2 RC1 passes.

Thanks everyone listed above who tried the release candidate and vote. Also, 
kudos to all who ever help with 2.8.2 release effort in all kinds of ways- 
especially the Yahoo! guys who deployed 2.8 in production environment and 
identify many issues with fixes. Also, Shane, Miklos and others to help with 
docker container effort during RC stage.

I'll push the release bits and send out an announcement for 2.8.2 soon.


Thanks,

Junping?


From: Eric Payne <erichadoo...@yahoo.com>
Sent: Tuesday, October 24, 2017 3:29 PM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

+1 (binding)

Thanks a lot, Junping!

I built and installed the source on a 6-node pseudo cluster. I simple sleep and 
streaming jobs that exercised intra-queue and inter-queue preemption, and used 
user weights.

-Eric


From: Junping Du <j...@hortonworks.com>
To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>; 
"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
Sent: Thursday, October 19, 2017 7:43 PM
Subject: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

Hi folks,
I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.

Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and will 
be the latest stable/production release for Apache Hadoop - it includes 315 new 
fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC1<http://home.apache.org/~junping_du/hadoop-2.8.2-RC0>

  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
66c47f2a01ad9637879e95f80c41f798373828fb

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org/> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1064<https://repository.apache.org/content/repositories/orgapachehadoop-1062>

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 10/24/2017 6pm PST time.

Thanks,

Junping

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Junping Du]

Allen,
 Do we have any solid evidence to show the HDFS unit tests going through 
the roof are due to serious memory leak by HDFS? Normally, I don't expect 
memory leak are identified in our UTs - mostly, it (test jvm gone) is just 
because of test or deployment issues. 
 Unless there is concrete evidence, my concern on seriously memory leak for 
HDFS on 2.8 is relatively low given some companies (Yahoo, Alibaba, etc.) have 
deployed 2.8 on large production environment for months. Non-serious memory 
leak (like forgetting to close stream in non-critical path, etc.) and other 
non-critical bugs always happens here and there that we have to live with.

Thanks,

Junping


From: Allen Wittenauer 
Sent: Tuesday, October 24, 2017 8:27 AM
To: Hadoop Common
Cc: Hdfs-dev; mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

> On Oct 23, 2017, at 12:50 PM, Allen Wittenauer  
> wrote:
>
>
>
> With no other information or access to go on, my current hunch is that one of 
> the HDFS unit tests is ballooning in memory size.  The easiest way to kill a 
> Linux machine is to eat all of the RAM, thanks to overcommit and that’s what 
> this “feels” like.
>
> Someone should verify if 2.8.2 has the same issues before a release goes out …


FWIW, I ran 2.8.2 last night and it has the same problems.

Also: the node didn’t die!  Looking through the workspace (so the next 
run will destroy them), two sets of logs stand out:

https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/ws/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

and

https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/ws/sourcedir/hadoop-hdfs-project/hadoop-hdfs/

It looks like my hunch is correct:  RAM in the HDFS unit tests are 
going through the roof.  It’s also interesting how MANY log files there are.  
Is surefire not picking up that jobs are dying?  Maybe not if memory is getting 
tight.

Anyway, at the point, branch-2.8 and higher are probably fubar’d. 
Additionally, I’ve filed YETUS-561 so that Yetus-controlled Docker containers 
can have their RAM limits set in order to prevent more nodes going catatonic.



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Junping Du]

Hi folks,
 I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.

 Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
will be the latest stable/production release for Apache Hadoop - it includes 
315 new fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical 
issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC1

  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
66c47f2a01ad9637879e95f80c41f798373828fb

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1064

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 10/24/2017 6pm PST time.

Thanks,

Junping

[Update] Apache Hadoop 2.8.2 Release Status

[Junping Du]

Hi folks,
 I would like to give you a quick update on 2.8.2 release status:

- First release candidate (RC0) is published over the last weekend, but several 
docker container blockers (bugs, documents, etc.)
 are reported so we decided to cancel the RC0 for vote.

- New coming release blockers (for docker container support) are YARN-7034 
(just committed), YARN-6623, YARN-6930 and YARN-7230. 
Shane, Miklos and Varun are actively working on this. Appreciate the effort 
here!

- I will kick off new release candidate (RC1) once these blockers are resolved.

To all committers, branch-2.8.2 is still open for blocker/critical issues 
landing, but for major/minor/trivial issues, please commit to branch-2.8 and 
marked the fixed version as 2.8.3.

Thanks all for heads up. Have a good weekend!


Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Tuesday, September 5, 2017 2:57 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

I assume the quiet over the holiday means we agreed to move forward without 
taking HADOOP-14439 into 2.8.2.
There is a new release building (docker based) issue could be related to 
HADOOP-14474 where we removed oracle java 7 installer due to recent download 
address/contract change by Oracle. The build refuse to work - report as 
JAVA_HOME issue, but hard coded my local java home in create-release or 
Dockerfile doesn't help so we may need to add java 7 installation back (no 
matter Oracle JDK 7 or openJDK 7).
Filed HADOOP-14842 with more details to track as blocker for 2.8.2.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Friday, September 1, 2017 12:37 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

This issue (HADOOP-14439) is out of my radar given it is marked as Minor 
priority. If my understanding is correct, here is a trade-off between security 
and backward compatibility. IMO, priority of security is generally higher than 
backward compatibility especially 2.8.0 is still non-production release.
I think we should skip this for 2.8.2 in case it doesn't break compatibility 
from 2.7.x. Thoughts?

Thanks,

Junping

From: larry mccay <lmc...@apache.org>
Sent: Friday, September 1, 2017 10:55 AM
To: Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

If we do "fix" this in 2.8.2 we should seriously consider not doing so in
3.0.
This is a very poor practice.

I can see an argument for backward compatibility in 2.8.x line though.

On Fri, Sep 1, 2017 at 1:41 PM, Steve Loughran <ste...@hortonworks.com>
wrote:

> One thing we need to consider is
>
> HADOOP-14439: regression: secret stripping from S3x URIs breaks some
> downstream code
>
> Hadoop 2.8 has a best-effort attempt to strip out secrets from the
> toString() value of an s3a or s3n path where someone has embedded them in
> the URI; this has caused problems in some uses, specifically: when people
> use secrets this way (bad) and assume that you can round trip paths to
> string and back
>
> Should we fix this? If so, Hadoop 2.8.2 is the time to do it
>
>
> > On 1 Sep 2017, at 11:14, Junping Du <j...@hortonworks.com> wrote:
> >
> > HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we
> are clean on blocker/critical issues now.
> > I finish practice of going through JACC report and no more incompatible
> public API changes get found between 2.8.2 and 2.7.4. Also I check commit
> history and fixed 10+ commits which are missing from branch-2.8.2 for some
> reason. So, the current branch-2.8.2 should be good to go for RC stage, and
> I will kick off our first RC tomorrow.
> > In the meanwhile, please don't land any commits to branch-2.8.2 since
> now. If some issues really belong to blocker, please ping me on the JIRA
> before doing any commits. branch-2.8 is still open for landing. Thanks for
> your cooperation!
> >
> >
> > Thanks,
> >
> > Junping
> >
> > 
> > From: Junping Du <j...@hortonworks.com>
> > Sent: Wednesday, August 30, 2017 12:35 AM
> > To: Brahma Reddy Battula; common-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> > Subject: Re: Apache Hadoop 2.8.2 Release Plan
> >
> > Thanks Brahma for co

[jira] [Created] (YARN-7230) Document DockerContainerRuntime for branch-2.8 with proper scope and claim as an experimental feature

[Junping Du (JIRA)]

Junping Du created YARN-7230:


 Summary: Document DockerContainerRuntime for branch-2.8 with 
proper scope and claim as an experimental feature
 Key: YARN-7230
 URL: https://issues.apache.org/jira/browse/YARN-7230
 Project: Hadoop YARN
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.8.1
Reporter: Junping Du
Priority: Blocker


YARN-5258 is to document new feature for docker container runtime which already 
get checked in trunk/branch-2. We need a similar one for branch-2.8. However, 
given we missed several patches, we need to define narrowed scope of these 
feature/improvements which match with existing patches landed in 2.8. Also, 
like YARN-6622, to document it as experimental.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Junping Du]

Miklos just ping me offline that a security fix should get landed to resolve a 
docker runtime issue. I will retrieval RC0 for security fixes landing.
In the mean while, if people here really think a document here is necessary 
(although not enough verification to work as an alpha feature) and can work out 
a patch soon, I am open to accept it. 

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Monday, September 11, 2017 5:32 PM
To: Daniel Templeton; Chris Douglas
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

Thanks Daniel for volunteering for documentation effort.
I suspect the problem we are facing here is not a simply documentation truncate 
effort, but indeed a incomplete feature problem. The umbrella JIRA YARN-3611 
has 31 issues are marked as resolved but only 9 patches get backport to 
branch-2.8 no matter intentioned or not. So far, I haven't heard anyone claimed 
that they are deploying/testing 2.8 release without cherry pick additional 
patches for enabling new docker executor runtime.
IMO, it is better to treat this feature in 2.8 as incomplete feature instead of 
an alpha/experimental feature, especially it is out of our previous 2.8 scope. 
If so, instead of documenting something misleading, we should keep new 
settings/configurations private (as it is now) to get rid of getting enabled by 
users occasionally.
Thoughts?


Thanks,

Junping

From: Daniel Templeton <dan...@cloudera.com>
Sent: Monday, September 11, 2017 4:45 PM
To: Chris Douglas; Junping Du
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

YARN-6622 is now committed to 2.9.  We could backport YARN-5258 and
YARN-6622 for 2.8, but it'll take some editing.  We'll have to check to
see what features are unsupported in 2.8 and remove those from the
docs.  Not a huge effort overall, though.  Probably a hour's work.  I
may have time to try do it later this week.  Anyone else want to volunteer?

Daniel

On 9/11/17 3:01 PM, Chris Douglas wrote:
> On Mon, Sep 11, 2017 at 2:52 PM, Junping Du <j...@hortonworks.com> wrote:
>> I don't think this -1 is reasonable, because:
>> - If you look at YARN-6622 closely, it targets to fix a problematic 
>> documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch 
>> only. It means it targets to fix a problem that 2.8.2 never exists.
> ...we're not going to document security implications- which include
> escalations to root- because we don't have _any_ documentation? Why
> don't we backport the documentation?
>
>> - New docker container support (replace of old DockerContainerExectutor) is 
>> still an alpha feature now which doesn't highlight in 2.8 major 
>> features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
>> adding documentation here is also not a blocker.
> YARN-6622 is *documenting* the fact that this is an alpha feature and
> that it shouldn't be enabled in secure environments. How are users
> supposed to make this determination without it?
>
>> Vote still continue until a real blocker comes.
> Soright. I remain -1. -C
>
>> ____________
>> From: Chris Douglas <cdoug...@apache.org>
>> Sent: Monday, September 11, 2017 12:00 PM
>> To: Junping Du
>> Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
>> mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; junping_du
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>
>> -1 (binding)
>>
>> I don't think we should release this without YARN-6622.
>>
>> Since this doesn't happen often: a -1 in this case is NOT a veto.
>> Releases are approved by majority vote of the PMC. -C
>>
>> On Mon, Sep 11, 2017 at 11:45 AM, Junping Du <j...@hortonworks.com> wrote:
>>> Thanks Mikols for notifying on this. I think docker support is general 
>>> known as alpha feature so document it as experimental is nice to have but 
>>> not a blocker for 2.8.2. I also noticed that our 2.7.x document 
>>> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>>>  without mentioning docker support is experimental. We may need to fix that 
>>> as well in following releases.
>>>
>>> I can also add it (mentioning docker container support feature is 
>>> experimental) to release message in public website just like previous 
>>> release we call 2.7.0/2.8.0 as non-production release.
>&

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Junping Du]

Thanks Daniel for volunteering for documentation effort. 
I suspect the problem we are facing here is not a simply documentation truncate 
effort, but indeed a incomplete feature problem. The umbrella JIRA YARN-3611 
has 31 issues are marked as resolved but only 9 patches get backport to 
branch-2.8 no matter intentioned or not. So far, I haven't heard anyone claimed 
that they are deploying/testing 2.8 release without cherry pick additional 
patches for enabling new docker executor runtime.
IMO, it is better to treat this feature in 2.8 as incomplete feature instead of 
an alpha/experimental feature, especially it is out of our previous 2.8 scope. 
If so, instead of documenting something misleading, we should keep new 
settings/configurations private (as it is now) to get rid of getting enabled by 
users occasionally.
Thoughts?


Thanks,

Junping

From: Daniel Templeton <dan...@cloudera.com>
Sent: Monday, September 11, 2017 4:45 PM
To: Chris Douglas; Junping Du
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

YARN-6622 is now committed to 2.9.  We could backport YARN-5258 and
YARN-6622 for 2.8, but it'll take some editing.  We'll have to check to
see what features are unsupported in 2.8 and remove those from the
docs.  Not a huge effort overall, though.  Probably a hour's work.  I
may have time to try do it later this week.  Anyone else want to volunteer?

Daniel

On 9/11/17 3:01 PM, Chris Douglas wrote:
> On Mon, Sep 11, 2017 at 2:52 PM, Junping Du <j...@hortonworks.com> wrote:
>> I don't think this -1 is reasonable, because:
>> - If you look at YARN-6622 closely, it targets to fix a problematic 
>> documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch 
>> only. It means it targets to fix a problem that 2.8.2 never exists.
> ...we're not going to document security implications- which include
> escalations to root- because we don't have _any_ documentation? Why
> don't we backport the documentation?
>
>> - New docker container support (replace of old DockerContainerExectutor) is 
>> still an alpha feature now which doesn't highlight in 2.8 major 
>> features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
>> adding documentation here is also not a blocker.
> YARN-6622 is *documenting* the fact that this is an alpha feature and
> that it shouldn't be enabled in secure environments. How are users
> supposed to make this determination without it?
>
>> Vote still continue until a real blocker comes.
> Soright. I remain -1. -C
>
>> ____
>> From: Chris Douglas <cdoug...@apache.org>
>> Sent: Monday, September 11, 2017 12:00 PM
>> To: Junping Du
>> Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
>> mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; junping_du
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>
>> -1 (binding)
>>
>> I don't think we should release this without YARN-6622.
>>
>> Since this doesn't happen often: a -1 in this case is NOT a veto.
>> Releases are approved by majority vote of the PMC. -C
>>
>> On Mon, Sep 11, 2017 at 11:45 AM, Junping Du <j...@hortonworks.com> wrote:
>>> Thanks Mikols for notifying on this. I think docker support is general 
>>> known as alpha feature so document it as experimental is nice to have but 
>>> not a blocker for 2.8.2. I also noticed that our 2.7.x document 
>>> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>>>  without mentioning docker support is experimental. We may need to fix that 
>>> as well in following releases.
>>>
>>> I can also add it (mentioning docker container support feature is 
>>> experimental) to release message in public website just like previous 
>>> release we call 2.7.0/2.8.0 as non-production release.
>>>
>>> I think vote should continue until we could find a real blocker.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Junping
>>>
>>>
>>> 
>>> From: Miklos Szegedi <miklos.szeg...@cloudera.com>
>>> Sent: Monday, September 11, 2017 10:07 AM
>>> To: Mingliang Liu
>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
>>> yarn-dev@hadoop.apache.org; junping_du; Junping Du
>>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>>
>>> Hello Junping,
>>>
>>> Thank you for working on this. Shoul

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Junping Du]

I don't think this -1 is reasonable, because:
- If you look at YARN-6622 closely, it targets to fix a problematic 
documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch only. 
It means it targets to fix a problem that 2.8.2 never exists.
- New docker container support (replace of old DockerContainerExectutor) is 
still an alpha feature now which doesn't highlight in 2.8 major 
features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
adding documentation here is also not a blocker.

Vote still continue until a real blocker comes.

Thanks,

Junping


From: Chris Douglas <cdoug...@apache.org>
Sent: Monday, September 11, 2017 12:00 PM
To: Junping Du
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

-1 (binding)

I don't think we should release this without YARN-6622.

Since this doesn't happen often: a -1 in this case is NOT a veto.
Releases are approved by majority vote of the PMC. -C

On Mon, Sep 11, 2017 at 11:45 AM, Junping Du <j...@hortonworks.com> wrote:
> Thanks Mikols for notifying on this. I think docker support is general known 
> as alpha feature so document it as experimental is nice to have but not a 
> blocker for 2.8.2. I also noticed that our 2.7.x document 
> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>  without mentioning docker support is experimental. We may need to fix that 
> as well in following releases.
>
> I can also add it (mentioning docker container support feature is 
> experimental) to release message in public website just like previous release 
> we call 2.7.0/2.8.0 as non-production release.
>
> I think vote should continue until we could find a real blocker.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Miklos Szegedi <miklos.szeg...@cloudera.com>
> Sent: Monday, September 11, 2017 10:07 AM
> To: Mingliang Liu
> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
> yarn-dev@hadoop.apache.org; junping_du; Junping Du
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>
> Hello Junping,
>
> Thank you for working on this. Should not YARN-6622 be addressed first? 
> "Summary: Document Docker work as experimental".
>
> Thank you,
> Miklos
>
>
> On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
> <lium...@gmail.com<mailto:lium...@gmail.com>> wrote:
> Thanks Junping for doing this!
>
> +1 (non-binding)
>
> - Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
> - Build package using maven (skipping tests) with Java 8
> - Spin up a test cluster in Docker containers having 1 master node (NN/RM) 
> and 3 slave nodes (DN/NM)
> - Operate the basic HDFS/YARN operations from command line, both client and 
> admin
> - Check NN/RM Web UI
> - Run distcp to copy files from/to local and HDFS
> - Run hadoop mapreduce examples: grep and wordcount
> - Check the HDFS service logs
>
> All looked good to me.
>
> Mingliang
>
>> On Sep 10, 2017, at 5:00 PM, Junping Du 
>> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
>>
>> Hi folks,
>> With fix of HADOOP-14842 get in, I've created our first release 
>> candidate (RC0) for Apache Hadoop 2.8.2.
>>
>> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
>> will be the latest stable/production release for Apache Hadoop - it includes 
>> 305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
>> issues.
>>
>>  More information about the 2.8.2 release plan can be found here: 
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>
>>  New RC is available at: 
>> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>>
>>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
>> e6597fe3000b06847d2bf55f2bab81770f4b2505
>>
>>  The maven artifacts are available via 
>> repository.apache.org<http://repository.apache.org> at: 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>>
>>  Please try the release and vote; the vote will run for the usual 5 
>> days, ending on 09/15/2017 5pm PST time.
>>
>> Thanks,
>>
>> Junping
>>
>
>
> -
> To unsubscribe, e-mail: 
> mapreduce-dev-unsubscr...@hadoop.apache.org<mailto:mapreduce-dev-unsubscr...@hadoop.apache.org>
> For additional commands, e-mail: 
&

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Junping Du]

Thanks Mikols for notifying on this. I think docker support is general known as 
alpha feature so document it as experimental is nice to have but not a blocker 
for 2.8.2. I also noticed that our 2.7.x document 
(https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
 without mentioning docker support is experimental. We may need to fix that as 
well in following releases.

I can also add it (mentioning docker container support feature is experimental) 
to release message in public website just like previous release we call 
2.7.0/2.8.0 as non-production release.

I think vote should continue until we could find a real blocker.


Thanks,


Junping



From: Miklos Szegedi <miklos.szeg...@cloudera.com>
Sent: Monday, September 11, 2017 10:07 AM
To: Mingliang Liu
Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; junping_du; Junping Du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

Hello Junping,

Thank you for working on this. Should not YARN-6622 be addressed first? 
"Summary: Document Docker work as experimental".

Thank you,
Miklos


On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
<lium...@gmail.com<mailto:lium...@gmail.com>> wrote:
Thanks Junping for doing this!

+1 (non-binding)

- Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
- Build package using maven (skipping tests) with Java 8
- Spin up a test cluster in Docker containers having 1 master node (NN/RM) and 
3 slave nodes (DN/NM)
- Operate the basic HDFS/YARN operations from command line, both client and 
admin
- Check NN/RM Web UI
- Run distcp to copy files from/to local and HDFS
- Run hadoop mapreduce examples: grep and wordcount
- Check the HDFS service logs

All looked good to me.

Mingliang

> On Sep 10, 2017, at 5:00 PM, Junping Du 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
>
> Hi folks,
> With fix of HADOOP-14842 get in, I've created our first release candidate 
> (RC0) for Apache Hadoop 2.8.2.
>
> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
> will be the latest stable/production release for Apache Hadoop - it includes 
> 305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
> issues.
>
>  More information about the 2.8.2 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>  New RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>
>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
> e6597fe3000b06847d2bf55f2bab81770f4b2505
>
>  The maven artifacts are available via 
> repository.apache.org<http://repository.apache.org> at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>
>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 09/15/2017 5pm PST time.
>
> Thanks,
>
> Junping
>


-
To unsubscribe, e-mail: 
mapreduce-dev-unsubscr...@hadoop.apache.org<mailto:mapreduce-dev-unsubscr...@hadoop.apache.org>
For additional commands, e-mail: 
mapreduce-dev-h...@hadoop.apache.org<mailto:mapreduce-dev-h...@hadoop.apache.org>

[VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Junping Du]

Hi folks,
 With fix of HADOOP-14842 get in, I've created our first release candidate 
(RC0) for Apache Hadoop 2.8.2.

 Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
will be the latest stable/production release for Apache Hadoop - it includes 
305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC0

  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
e6597fe3000b06847d2bf55f2bab81770f4b2505

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1062

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 09/15/2017 5pm PST time.

Thanks,

Junping

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

I assume the quiet over the holiday means we agreed to move forward without 
taking HADOOP-14439 into 2.8.2.
There is a new release building (docker based) issue could be related to 
HADOOP-14474 where we removed oracle java 7 installer due to recent download 
address/contract change by Oracle. The build refuse to work - report as 
JAVA_HOME issue, but hard coded my local java home in create-release or 
Dockerfile doesn't help so we may need to add java 7 installation back (no 
matter Oracle JDK 7 or openJDK 7). 
Filed HADOOP-14842 with more details to track as blocker for 2.8.2.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Friday, September 1, 2017 12:37 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

This issue (HADOOP-14439) is out of my radar given it is marked as Minor 
priority. If my understanding is correct, here is a trade-off between security 
and backward compatibility. IMO, priority of security is generally higher than 
backward compatibility especially 2.8.0 is still non-production release.
I think we should skip this for 2.8.2 in case it doesn't break compatibility 
from 2.7.x. Thoughts?

Thanks,

Junping

From: larry mccay <lmc...@apache.org>
Sent: Friday, September 1, 2017 10:55 AM
To: Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

If we do "fix" this in 2.8.2 we should seriously consider not doing so in
3.0.
This is a very poor practice.

I can see an argument for backward compatibility in 2.8.x line though.

On Fri, Sep 1, 2017 at 1:41 PM, Steve Loughran <ste...@hortonworks.com>
wrote:

> One thing we need to consider is
>
> HADOOP-14439: regression: secret stripping from S3x URIs breaks some
> downstream code
>
> Hadoop 2.8 has a best-effort attempt to strip out secrets from the
> toString() value of an s3a or s3n path where someone has embedded them in
> the URI; this has caused problems in some uses, specifically: when people
> use secrets this way (bad) and assume that you can round trip paths to
> string and back
>
> Should we fix this? If so, Hadoop 2.8.2 is the time to do it
>
>
> > On 1 Sep 2017, at 11:14, Junping Du <j...@hortonworks.com> wrote:
> >
> > HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we
> are clean on blocker/critical issues now.
> > I finish practice of going through JACC report and no more incompatible
> public API changes get found between 2.8.2 and 2.7.4. Also I check commit
> history and fixed 10+ commits which are missing from branch-2.8.2 for some
> reason. So, the current branch-2.8.2 should be good to go for RC stage, and
> I will kick off our first RC tomorrow.
> > In the meanwhile, please don't land any commits to branch-2.8.2 since
> now. If some issues really belong to blocker, please ping me on the JIRA
> before doing any commits. branch-2.8 is still open for landing. Thanks for
> your cooperation!
> >
> >
> > Thanks,
> >
> > Junping
> >
> > 
> > From: Junping Du <j...@hortonworks.com>
> > Sent: Wednesday, August 30, 2017 12:35 AM
> > To: Brahma Reddy Battula; common-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> > Subject: Re: Apache Hadoop 2.8.2 Release Plan
> >
> > Thanks Brahma for comment on this thread. To be clear, I always update
> branch version just before RC kicking off.
> >
> > For 2.8.2 release, I don't have plan to involve big top or other
> third-party test tools. As always, we will rely on test/verify efforts from
> community especially from large deployed production cluster - as far as I
> know,  there are already several companies. like: Yahoo!, Alibaba, etc.
> already deploy 2.8 release in large production clusters for months which
> give me more confidence on 2.8.2.
> >
> >
> > Here is more update on 2.8.2 release:
> >
> > Blocker issues:
> >
> >-  A new blocker YARN-7076 get reported and fixed by Jian He through
> last weekend.
> >
> >-  Another new blocker - HADOOP-14814 get identified from my latest
> jdiff run against 2.7.4. The simple fix on an incompatible API change
> should get commit soon.
> >
> >
> > Critical issues:
> >
> >-  YARN-7083 already get committed. Thanks Jason for reporting the
> issue and delivering the fix.
> >
> >-  Y

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

This issue (HADOOP-14439) is out of my radar given it is marked as Minor 
priority. If my understanding is correct, here is a trade-off between security 
and backward compatibility. IMO, priority of security is generally higher than 
backward compatibility especially 2.8.0 is still non-production release. 
I think we should skip this for 2.8.2 in case it doesn't break compatibility 
from 2.7.x. Thoughts?

Thanks,

Junping

From: larry mccay <lmc...@apache.org>
Sent: Friday, September 1, 2017 10:55 AM
To: Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

If we do "fix" this in 2.8.2 we should seriously consider not doing so in
3.0.
This is a very poor practice.

I can see an argument for backward compatibility in 2.8.x line though.

On Fri, Sep 1, 2017 at 1:41 PM, Steve Loughran <ste...@hortonworks.com>
wrote:

> One thing we need to consider is
>
> HADOOP-14439: regression: secret stripping from S3x URIs breaks some
> downstream code
>
> Hadoop 2.8 has a best-effort attempt to strip out secrets from the
> toString() value of an s3a or s3n path where someone has embedded them in
> the URI; this has caused problems in some uses, specifically: when people
> use secrets this way (bad) and assume that you can round trip paths to
> string and back
>
> Should we fix this? If so, Hadoop 2.8.2 is the time to do it
>
>
> > On 1 Sep 2017, at 11:14, Junping Du <j...@hortonworks.com> wrote:
> >
> > HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we
> are clean on blocker/critical issues now.
> > I finish practice of going through JACC report and no more incompatible
> public API changes get found between 2.8.2 and 2.7.4. Also I check commit
> history and fixed 10+ commits which are missing from branch-2.8.2 for some
> reason. So, the current branch-2.8.2 should be good to go for RC stage, and
> I will kick off our first RC tomorrow.
> > In the meanwhile, please don't land any commits to branch-2.8.2 since
> now. If some issues really belong to blocker, please ping me on the JIRA
> before doing any commits. branch-2.8 is still open for landing. Thanks for
> your cooperation!
> >
> >
> > Thanks,
> >
> > Junping
> >
> > 
> > From: Junping Du <j...@hortonworks.com>
> > Sent: Wednesday, August 30, 2017 12:35 AM
> > To: Brahma Reddy Battula; common-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> > Subject: Re: Apache Hadoop 2.8.2 Release Plan
> >
> > Thanks Brahma for comment on this thread. To be clear, I always update
> branch version just before RC kicking off.
> >
> > For 2.8.2 release, I don't have plan to involve big top or other
> third-party test tools. As always, we will rely on test/verify efforts from
> community especially from large deployed production cluster - as far as I
> know,  there are already several companies. like: Yahoo!, Alibaba, etc.
> already deploy 2.8 release in large production clusters for months which
> give me more confidence on 2.8.2.
> >
> >
> > Here is more update on 2.8.2 release:
> >
> > Blocker issues:
> >
> >-  A new blocker YARN-7076 get reported and fixed by Jian He through
> last weekend.
> >
> >-  Another new blocker - HADOOP-14814 get identified from my latest
> jdiff run against 2.7.4. The simple fix on an incompatible API change
> should get commit soon.
> >
> >
> > Critical issues:
> >
> >-  YARN-7083 already get committed. Thanks Jason for reporting the
> issue and delivering the fix.
> >
> >-  YARN-6091 get push out from 2.8.2 as issue is not a regression and
> pending for a while.
> >
> >-  Daryn is actively working on HADOOP-9747 for a while, and the
> patch are getting close to be committed. However, according to Daryn, the
> patch seems to cause some regression in some corner cases in secured
> environment (Storm auto tgt, etc.). May need some additional watch/review
> on this JIRA's fixes.
> >
> >
> >
> > My monitoring JACC report between 2.8.2 and 2.7.4 will get finish
> tomorrow. If everything is going smoothly, I am planning to kick off RC0
> around holiday (this weekend).
> >
> >
> >
> > Thanks,
> >
> >
> >
> > ​Junping
> >
> >
> >
> > 
> > From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
> > Sent: Tuesday, A

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we are 
clean on blocker/critical issues now.
I finish practice of going through JACC report and no more incompatible public 
API changes get found between 2.8.2 and 2.7.4. Also I check commit history and 
fixed 10+ commits which are missing from branch-2.8.2 for some reason. So, the 
current branch-2.8.2 should be good to go for RC stage, and I will kick off our 
first RC tomorrow.
In the meanwhile, please don't land any commits to branch-2.8.2 since now. If 
some issues really belong to blocker, please ping me on the JIRA before doing 
any commits. branch-2.8 is still open for landing. Thanks for your cooperation!


Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, August 30, 2017 12:35 AM
To: Brahma Reddy Battula; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Thanks Brahma for comment on this thread. To be clear, I always update branch 
version just before RC kicking off.

For 2.8.2 release, I don't have plan to involve big top or other third-party 
test tools. As always, we will rely on test/verify efforts from community 
especially from large deployed production cluster - as far as I know,  there 
are already several companies. like: Yahoo!, Alibaba, etc. already deploy 2.8 
release in large production clusters for months which give me more confidence 
on 2.8.2.


Here is more update on 2.8.2 release:

Blocker issues:

-  A new blocker YARN-7076 get reported and fixed by Jian He through last 
weekend.

-  Another new blocker - HADOOP-14814 get identified from my latest jdiff 
run against 2.7.4. The simple fix on an incompatible API change should get 
commit soon.


Critical issues:

-  YARN-7083 already get committed. Thanks Jason for reporting the issue 
and delivering the fix.

-  YARN-6091 get push out from 2.8.2 as issue is not a regression and 
pending for a while.

-  Daryn is actively working on HADOOP-9747 for a while, and the patch are 
getting close to be committed. However, according to Daryn, the patch seems to 
cause some regression in some corner cases in secured environment (Storm auto 
tgt, etc.). May need some additional watch/review on this JIRA's fixes.



My monitoring JACC report between 2.8.2 and 2.7.4 will get finish tomorrow. If 
everything is going smoothly, I am planning to kick off RC0 around holiday 
(this weekend).



Thanks,



​Junping




From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Tuesday, August 29, 2017 8:42 AM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Hi All

Update on 2.8.2 release status
we are down to 3 critical issues ( YARN-6091,YARN-7083,HADOOP-9747),all are 
patch available and closer to commit.
Junping is closing tracking this.

Todo:

1) Update pom.xml ..?  currently it's with 2.8.3
https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21
<https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21>2) Wiki is 
outdated, need to update the wiki..?
3) As this is going to stable release,are we planing enable Big top for 2.8.2 
testing Or Dynamometer testing (anybody from linked-in can help)..?

@Junping Du<mailto:j...@hortonworks.com>,Please correct me,if I am wrong.


--Brahma Reddy Battula
____________
From: Junping Du <j...@hortonworks.com>
Sent: Monday, August 7, 2017 2:44 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x
Issue Navigator - ASF JIRA<https://s.apache.org/JM5x>
s.apache.org
Linked Applications. Loading… Dashboards




I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping
____________
From: Junping Du <j...@hortonworks.com>
Sent: Monday, July 24, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and lo

[jira] [Created] (YARN-7138) Fix incompatible API change for YarnScheduler involved by YARN-5521

[Junping Du (JIRA)]

Junping Du created YARN-7138:


 Summary: Fix incompatible API change for YarnScheduler involved by 
YARN-5521
 Key: YARN-7138
 URL: https://issues.apache.org/jira/browse/YARN-7138
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Reporter: Junping Du
Priority: Blocker


>From JACC report for 2.8.2 against 2.7.4, it indicates that we have 
>incompatible changes happen in YarnScheduler:
{noformat}
hadoop-yarn-server-resourcemanager-2.7.4.jar, YarnScheduler.class
package org.apache.hadoop.yarn.server.resourcemanager.scheduler
YarnScheduler.allocate ( ApplicationAttemptId p1, List p2, 
List p3, List p4, List p5 ) [abstract]  :  
Allocation 
{noformat}
The root cause is YARN-5221. We should change it back or workaround this by 
adding back original API (mark as deprecated if not used any more).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Thanks Brahma for comment on this thread. To be clear, I always update branch 
version just before RC kicking off.

For 2.8.2 release, I don't have plan to involve big top or other third-party 
test tools. As always, we will rely on test/verify efforts from community 
especially from large deployed production cluster - as far as I know,  there 
are already several companies. like: Yahoo!, Alibaba, etc. already deploy 2.8 
release in large production clusters for months which give me more confidence 
on 2.8.2.


Here is more update on 2.8.2 release:

Blocker issues:

-  A new blocker YARN-7076 get reported and fixed by Jian He through last 
weekend.

-  Another new blocker - HADOOP-14814 get identified from my latest jdiff 
run against 2.7.4. The simple fix on an incompatible API change should get 
commit soon.


Critical issues:

-  YARN-7083 already get committed. Thanks Jason for reporting the issue 
and delivering the fix.

-  YARN-6091 get push out from 2.8.2 as issue is not a regression and 
pending for a while.

-  Daryn is actively working on HADOOP-9747 for a while, and the patch are 
getting close to be committed. However, according to Daryn, the patch seems to 
cause some regression in some corner cases in secured environment (Storm auto 
tgt, etc.). May need some additional watch/review on this JIRA's fixes.



My monitoring JACC report between 2.8.2 and 2.7.4 will get finish tomorrow. If 
everything is going smoothly, I am planning to kick off RC0 around holiday 
(this weekend).



Thanks,



​Junping




From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Tuesday, August 29, 2017 8:42 AM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Hi All

Update on 2.8.2 release status
we are down to 3 critical issues ( YARN-6091,YARN-7083,HADOOP-9747),all are 
patch available and closer to commit.
Junping is closing tracking this.

Todo:

1) Update pom.xml ..?  currently it's with 2.8.3
https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21
<https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21>2) Wiki is 
outdated, need to update the wiki..?
3) As this is going to stable release,are we planing enable Big top for 2.8.2 
testing Or Dynamometer testing (anybody from linked-in can help)..?

@Junping Du<mailto:j...@hortonworks.com>,Please correct me,if I am wrong.


--Brahma Reddy Battula
________
From: Junping Du <j...@hortonworks.com>
Sent: Monday, August 7, 2017 2:44 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x
Issue Navigator - ASF JIRA<https://s.apache.org/JM5x>
s.apache.org
Linked Applications. Loading… Dashboards




I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping
________
From: Junping Du <j...@hortonworks.com>
Sent: Monday, July 24, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping


____________
From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behav

[jira] [Created] (YARN-7124) CLONE - Log aggregation deletes/renames while file is open

[Junping Du (JIRA)]

Junping Du created YARN-7124:


 Summary: CLONE - Log aggregation deletes/renames while file is open
 Key: YARN-7124
 URL: https://issues.apache.org/jira/browse/YARN-7124
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.8.2
Reporter: Daryn Sharp
Assignee: Jason Lowe
Priority: Critical


YARN-6288 changes the log aggregation writer to be an autoclosable.  
Unfortunately the try-with-resources block for the writer will either rename or 
delete the log while open.

Assuming the NM's behavior is correct, deleting open files only results in 
ominous WARNs in the nodemanager log and increases the rate of logging in the 
NN when the implicit try-with-resource close fails.  These red herrings 
complicate debugging efforts.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7027) Log aggregation finish time should get logged for trouble shooting.

[Junping Du (JIRA)]

Junping Du created YARN-7027:


 Summary: Log aggregation finish time should get logged for trouble 
shooting.
 Key: YARN-7027
 URL: https://issues.apache.org/jira/browse/YARN-7027
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: log-aggregation
Reporter: Junping Du
Assignee: Junping Du


Now, RM track application log aggregation status in RMApp and the status change 
is triggered by NM heartbeat with log aggregation report. For each node's log 
aggregation status change from in-progress 
(NOT_START,RUNNING,RUNNING_WITH_FAILURE) to final status (SUCCEEDED,FAILED, 
TIMEOUT), it trigger an aggregation for log aggregation status: 
updateLogAggregationStatus(). The whole progress is log less and we cannot 
trace the log aggregation problem (delay of log aggregation, etc.) from RM (or 
NM) log. We should add more log here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-1038) LocalizationProtocolPBClientImpl RPC failing

[Junping Du (JIRA)]

 [ 
https://issues-test.apache.org/jira/browse/YARN-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du resolved YARN-1038.
--
Resolution: Cannot Reproduce

I don't think trunk branch has this problem now, just resolve as cannot 
reproduce.

> LocalizationProtocolPBClientImpl RPC failing
> 
>
> Key: YARN-1038
> URL: https://issues-test.apache.org/jira/browse/YARN-1038
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Alejandro Abdelnur
>Priority: Blocker
>
> Trying to run an MR job in trunk is failing with:
> {code}
> 2013-08-06 22:24:21,498 WARN org.apache.hadoop.ipc.Client: interrupted 
> waiting to send rpc request to server
> java.lang.InterruptedException
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279)
>   at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>   at 
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1019)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1372)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1352)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy25.heartbeat(Unknown Source)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:250)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:164)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:107)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:977)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x

I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Monday, July 24, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping



From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If no other concerns, I will do the branch update in next 30 minutes.


Thanks,


Junping



From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Sunday, July 23, 2017 1:50 AM
To: Junping Du; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan



Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are 
missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two 
commits.Hope we'll not miss any commits which present in only in branch-2.8.2.

________
From: Junping Du <j...@hortonworks.com>
Sent: Saturday, July 22, 2017 5:57 AM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping

____________
From: Junping Du <j...@hortonworks.com>
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
>
> Thanks for suggestions, Jason and K

[jira] [Resolved] (YARN-6891) Can kill other user's applications via RM UI

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du resolved YARN-6891.
--
Resolution: Duplicate

> Can kill other user's applications via RM UI
> 
>
> Key: YARN-6891
> URL: https://issues.apache.org/jira/browse/YARN-6891
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Sumana Sathish
>        Assignee: Junping Du
>Priority: Critical
>
> In a  secured cluster with UI unsecured which has following config
> {code}
> "hadoop.http.authentication.simple.anonymous.allowed" => "true"
> "hadoop.http.authentication.type" => kerberos
> {code}
> UI can be accessed without any security setting.
> Also any user can kill other user's applications via UI



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6890) If UI is not secured, we allow user to kill other users' job even yarn cluster is secured.

[Junping Du (JIRA)]

Junping Du created YARN-6890:


 Summary: If UI is not secured, we allow user to kill other users' 
job even yarn cluster is secured.
 Key: YARN-6890
 URL: https://issues.apache.org/jira/browse/YARN-6890
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Sumana Sathish
Assignee: Junping Du
Priority: Critical


Configuring SPNEGO for web browser could be a head ache, so many production 
cluster choose to configure a unsecured UI access even for a secured cluster. 
In this setup, users (login as some random guy) could watch other users job 
which is expected. However, the kill button (added in YARN-3249 which enabled 
by default) shouldn't work in this situation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re:

[Junping Du]

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping



From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If no other concerns, I will do the branch update in next 30 minutes.


Thanks,


Junping



From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Sunday, July 23, 2017 1:50 AM
To: Junping Du; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan



Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are 
missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two 
commits.Hope we'll not miss any commits which present in only in branch-2.8.2.

________
From: Junping Du <j...@hortonworks.com>
Sent: Saturday, July 22, 2017 5:57 AM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping

________
From: Junping Du <j...@hortonworks.com>
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-dev@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If no other concerns, I will do the branch update in next 30 minutes.


Thanks,


Junping



From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Sunday, July 23, 2017 1:50 AM
To: Junping Du; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan



Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are 
missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two 
commits.Hope we'll not miss any commits which present in only in branch-2.8.2.

________
From: Junping Du <j...@hortonworks.com>
Sent: Saturday, July 22, 2017 5:57 AM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping

________
From: Junping Du <j...@hortonworks.com>
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-dev@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
>
> Jason
>
>
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
> <kih...@yahoo-inc.com.INVALID> wrote:
>
>
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
>
> On Thursday, July 20, 201

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-dev@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
>
> Jason
>
>
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
> <kih...@yahoo-inc.com.INVALID> wrote:
>
>
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
>
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du <j...@hortonworks.com> 
> wrote:
>
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
>
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
>
>  Please share thoughts if you have any. Thanks!
>
> Thanks,
>
> Junping
>
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
> 
> From: Vinod Kumar Vavilapalli <vino...@apache.org>
> Sent: Thursday, July 20, 2017 1:05 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released
>
> Hi all,
>
> The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
> page: http://hadoop.apache.org/releases.html#Download
> This is a sec

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-dev@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
>
> Jason
>
>
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
> <kih...@yahoo-inc.com.INVALID> wrote:
>
>
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
>
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du <j...@hortonworks.com> 
> wrote:
>
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
>
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
>
>  Please share thoughts if you have any. Thanks!
>
> Thanks,
>
> Junping
>
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
> 
> From: Vinod Kumar Vavilapalli <vino...@apache.org>
> Sent: Thursday, July 20, 2017 1:05 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released
>
> Hi all,
>
> The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
> page: http://hadoop.apache.org/releases.html#Download
> This is a security release in the 2.8.0 release line. It consists of 2.8.0 
> plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.
>
> Please note that 2.8.x release line continues to be not yet ready for 
> production use. Critical issues are being ironed out via testing and 
> downstream adoption. Production users should wait for a subsequent release in 
> the 2.8.x line.
>
> Thanks
> +Vinod
>
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Thanks for suggestions, Jason and Kihwal!
+1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
cannot be abandoned/replaced (suspect all branches are read-only now), I will 
manually merge all commits that not landed on 2.8.2 yet.

Thanks,

Junping

From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
Sent: Friday, July 21, 2017 8:17 AM
To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

+1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that are 
in branch-2.8.  There also are a lot of JIRAs that claim they are fixed in 
2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on recent 
activity in branch-2.8 would solve both of these issues, and we'd only need to 
move the handful of JIRAs that have marked themselves correctly as fixed in 
2.8.3 to be fixed in 2.8.2.

Jason


On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
<kih...@yahoo-inc.com.INVALID> wrote:


 Thanks for driving the next 2.8 release, Junping. While I was committing a 
blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
2.8.2.
Thanks,Kihwal

On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du <j...@hortonworks.com> 
wrote:

Hi all,
Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
released today which is a special security release. Now, we should work towards 
2.8.2 release which aim for production deployment. The focus obviously is to 
fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We 
currently have 13 blocker/critical issues, and 10 of them are Patch Available.

  I plan to cut an RC in a month - target for releasing before end of Aug., to 
give enough time for outstanding blocker / critical issues. Will start moving 
out any tickets that are not blockers and/or won't fit the timeline. For 
progress of releasing effort, please refer our release wiki [2].

  Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
[2] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Thursday, July 20, 2017 1:05 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released

Hi all,

The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
page: http://hadoop.apache.org/releases.html#Download
This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus 
security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.

Please note that 2.8.x release line continues to be not yet ready for 
production use. Critical issues are being ironed out via testing and downstream 
adoption. Production users should wait for a subsequent release in the 2.8.x 
line.

Thanks
+Vinod


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Hi all,
 Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
released today which is a special security release. Now, we should work towards 
2.8.2 release which aim for production deployment. The focus obviously is to 
fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We 
currently have 13 blocker/critical issues, and 10 of them are Patch Available.

   I plan to cut an RC in a month - target for releasing before end of Aug., to 
give enough time for outstanding blocker / critical issues. Will start moving 
out any tickets that are not blockers and/or won't fit the timeline. For 
progress of releasing effort, please refer our release wiki [2].

   Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
[2] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


From: Vinod Kumar Vavilapalli 
Sent: Thursday, July 20, 2017 1:05 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released

Hi all,

The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
page: http://hadoop.apache.org/releases.html#Download
This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus 
security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.

Please note that 2.8.x release line continues to be not yet ready for 
production use. Critical issues are being ironed out via testing and downstream 
adoption. Production users should wait for a subsequent release in the 2.8.x 
line.

Thanks
+Vinod


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: 2.8 Release activities

[Junping Du]

Haohui,
I am waiting for a special security release on branch-2.8 get out to resume 
the release work for production release of 2.8. You should be on security alias 
and ask for update there.

Thanks,

Junping


From: Haohui Mai 
Sent: Wednesday, July 12, 2017 11:48 PM
To: Hadoop Common; yarn-dev@hadoop.apache.org; Hdfs-dev
Subject: 2.8 Release activities

Hi,

Just curious -- what is the current status of the 2.8 release? It looks
like the release process for some time.

There are 5 or 6 blocker / critical bugs of the upcoming 2.8 release:

https://issues.apache.org/jira/browse/YARN-6654?jql=project%20in%20(HDFS%2C%20HADOOP%2C%20MAPREDUCE%2C%20YARN)%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20%22Target%20Version%2Fs%22%20in%20(2.8.2%2C%202.8.3)

I think we can address them in reasonable amount of effort.

We are interested in putting 2.8.x in production and it would be great to
have a maintenance Apache release for the 2.8 line.

I wonder, are there any concerns of not getting the release out? We might
be able to get some helps internally to fix the issues in the 2.8 lines. I
can also volunteer to be the release manager for 2.8.2 if it requires more
effort to coordinate to push the release out.

Regards,
Haohui



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-5007) Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du resolved YARN-5007.
--
Resolution: Later
  Assignee: (was: Andras Bokor)

> Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster
> ---
>
> Key: YARN-5007
> URL: https://issues.apache.org/jira/browse/YARN-5007
> Project: Hadoop YARN
>  Issue Type: Test
>Reporter: Andras Bokor
>  Labels: oct16-easy
> Attachments: YARN-5007.01.patch, YARN-5007.02.patch, 
> YARN-5007.03.patch
>
>
> MiniYarnCluster has a deprecated constructor which is called by the other 
> constructors and it causes javac warnings during the build.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6534) ResourceManager failed due to TimelineClient try to init SSLFactory even https is not enabled

[Junping Du (JIRA)]

Junping Du created YARN-6534:


 Summary: ResourceManager failed due to TimelineClient try to init 
SSLFactory even https is not enabled
 Key: YARN-6534
 URL: https://issues.apache.org/jira/browse/YARN-6534
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 3.0.0-alpha3
Reporter: Junping Du
Priority: Blocker


In a non-secured cluster, RM get failed consistently due to 
TimelineServiceV1Publisher tries to init TimelineClient with SSLFactory without 
any checking on if https get used.

{noformat}
2017-04-26 21:09:10,683 FATAL resourcemanager.ResourceManager 
(ResourceManager.java:main(1457)) - Error starting ResourceManager
org.apache.hadoop.service.ServiceStateException: java.io.FileNotFoundException: 
/etc/security/clientKeys/all.jks (No such file or directory)
at 
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:131)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at 
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractSystemMetricsPublisher.serviceInit(AbstractSystemMetricsPublisher.java:59)
at 
org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher.serviceInit(TimelineServiceV1Publisher.java:67)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:344)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1453)
Caused by: java.io.FileNotFoundException: /etc/security/clientKeys/all.jks (No 
such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at 
org.apache.hadoop.security.ssl.ReloadingX509TrustManager.loadTrustManager(ReloadingX509TrustManager.java:168)
at 
org.apache.hadoop.security.ssl.ReloadingX509TrustManager.(ReloadingX509TrustManager.java:86)
at 
org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:219)
at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:179)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineConnector.getSSLFactory(TimelineConnector.java:176)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineConnector.serviceInit(TimelineConnector.java:106)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 11 more
{noformat}
CC [~rohithsharma] and [~gtCarrera9]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.1 Release Plan

[Junping Du]

I just cut off branch-2.8.1 from branch-2.8 for 2.8.1 release. So from now, all 
patch target for 2.8.1 need to be checked into branch-2.8.1 in addition to 
branch-2.8.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Monday, April 03, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Apache Hadoop 2.8.1 Release Plan

Hi all,
   We just released Apache Hadoop 2.8.0 recently [1] but it is not for 
production yet due to some issues identified. Now, we should work towards 2.8.1 
release which aim for production deployment. The focus obviously is to fix 
blocker/critical issues [2], bug-fixes and *no* features / improvements.

   I plan to cut an RC in a month - target for releasing at mid of May, to give 
enough time for outstanding blocker / critical issues. Will start moving out 
any tickets that are not blockers and/or won't fit the timeline - there are 2 
blockers and 9 critical tickets outstanding as of now. For progress of 
releasing effort, please refer our release wiki [3].

   Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.0 release announcement: 
http://www.mail-archive.com/general@hadoop.apache.org/msg07443.html
[2] 2.8.1 release Blockers/Criticals: https://s.apache.org/KGxC
[3] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Upgrading minimum version of Maven to 3.1 from 3.0

[Junping Du]

Thanks Sunil to bring it up. However, any reason not to go to latest version of 
maven (3.3+)? Even version 3.1 is quite old - released 3+ years from now 
(please refer: https://archive.apache.org/dist/maven/binaries/).
btw, I assume we only plan to change maven version on trunk given new YARN UI 
only merge to trunk branch. Isn't it?

Thanks,

Junping

From: Akira Ajisaka 
Sent: Monday, April 03, 2017 10:02 PM
To: Sunil G; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: Upgrading minimum version of Maven to 3.1 from 3.0

As I mentioned in YARN-6421. I'm +1 for upgrading to 3.1+ because
the latest version of Maven 3.0.x is quite old. (4 years ago)

We need to update dev-support/docker/Dockerfile to enable Maven 3.1+ for
precommit Jenkins job.

Regards,
Akira

On 2017/04/03 18:49, Sunil G wrote:
> Hi Folks,
>
> Recently we were doing build framework up-gradation for Yarn Ui. In order
> to compile yarn-ui on various architectures, we were using
> frontend-maven-plugin 0.0.22 version.
> However build is failing in *ppc64le.* If we could use latest version of
> frontend-maven-plugin, we could resolve this error. (such as using 1.1
> version). But this requires maven version 3.1 minimum. YARN-6421 is
> tracking this issue, and we thought we can propose to upgrade to maven 3.1
>
> Kindly share your thoughts.
>
> Thanks
> + Sunil
>

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop 2.8.1 Release Plan

[Junping Du]

Hi all,
   We just released Apache Hadoop 2.8.0 recently [1] but it is not for 
production yet due to some issues identified. Now, we should work towards 2.8.1 
release which aim for production deployment. The focus obviously is to fix 
blocker/critical issues [2], bug-fixes and *no* features / improvements.

   I plan to cut an RC in a month - target for releasing at mid of May, to give 
enough time for outstanding blocker / critical issues. Will start moving out 
any tickets that are not blockers and/or won't fit the timeline - there are 2 
blockers and 9 critical tickets outstanding as of now. For progress of 
releasing effort, please refer our release wiki [3].

   Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.0 release announcement: 
http://www.mail-archive.com/general@hadoop.apache.org/msg07443.html
[2] 2.8.1 release Blockers/Criticals: https://s.apache.org/KGxC
[3] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Quick update: the missing bits in svn release directories (reported in 
https://issues.apache.org/jira/browse/INFRA-13749) already get resolved with 
workaround of bypassing 200M file size limitation. We should document this in 
our release wiki.
However, a new issue is found that releasing bits are missing from mirror sites 
which get reported in https://issues.apache.org/jira/browse/INFRA-13755. The 
Apache infrastructure team is investigating right now. Will send release news 
when issue get fixed. Stay tuned!

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Friday, March 24, 2017 6:21 PM
To: Allen Wittenauer
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Thanks Allen, I already add it back in JIRA's 2.8.0 release date.
However, I met another issue in uploading our 2.8.0 release bits to SVN: 
hadoop-2.8.0.tar.gz get uploaded failed with following exception:

Adding  (bin)  hadoop-2.8.0/hadoop-2.8.0.tar.gz
Transmitting file data .svn: E175002: Commit failed (details follow):
svn: E175002: PUT request on 
'/repos/dist/!svn/txr/18902-g1s/release/hadoop/common/hadoop-2.8.0/hadoop-2.8.0.tar.gz'
 failed

Other (smaller) files get uploaded successfully. Once I suspected it could due 
to no space, so I removed some old release bits (like: 2.5.2) but still no 
luck. So may be the size of hadoop-2.8.0.tar.gz become a problem here?
I already reported the issue in 
https://issues.apache.org/jira/browse/INFRA-13749, but will appreciate if 
someone got idea on how to get through this.

Thanks,

Junping

From: Allen Wittenauer <a...@effectivemachines.com>
Sent: Thursday, March 23, 2017 5:27 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Just a heads up.  Looks like some removed the Finish Date off of 2.8.0 in JIRA. 
 It needs to be put back to match what is in the artifacts that we voted on.

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Thanks Allen, I already add it back in JIRA's 2.8.0 release date.
However, I met another issue in uploading our 2.8.0 release bits to SVN: 
hadoop-2.8.0.tar.gz get uploaded failed with following exception:

Adding  (bin)  hadoop-2.8.0/hadoop-2.8.0.tar.gz
Transmitting file data .svn: E175002: Commit failed (details follow):
svn: E175002: PUT request on 
'/repos/dist/!svn/txr/18902-g1s/release/hadoop/common/hadoop-2.8.0/hadoop-2.8.0.tar.gz'
 failed

Other (smaller) files get uploaded successfully. Once I suspected it could due 
to no space, so I removed some old release bits (like: 2.5.2) but still no 
luck. So may be the size of hadoop-2.8.0.tar.gz become a problem here? 
I already reported the issue in 
https://issues.apache.org/jira/browse/INFRA-13749, but will appreciate if 
someone got idea on how to get through this.

Thanks,

Junping

From: Allen Wittenauer <a...@effectivemachines.com>
Sent: Thursday, March 23, 2017 5:27 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Just a heads up.  Looks like some removed the Finish Date off of 2.8.0 in JIRA. 
 It needs to be put back to match what is in the artifacts that we voted on.

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Thanks again for all who verified and voted!


I give my binding +1 to conclude the vote for 2.8.0 RC3, based on:
- Build from source and verify signatures
- Deploy pseudo-distributed cluster with capacity scheduler
- Verify UI of daemons, like: NameNode, ResourceManager, NodeManager, etc.
- Run some example MR jobs, like: PI, sleep, etc.

Now, we have:

7 binding +1s, from:
 Wangda Tan, Jason Lowe, Akira Ajisaka, Ravi Prakash,
 Karthik Kambatla, Jian He, Junping Du

18 non-binding +1s, from:
Miklos Szegedi, Eric Payne, Daniel Templeton, Mingliang Liu,
Sunil Govind, Marton Elek, Brahma Reddy Battula, Masatake Iwasaki,
Gergo Pasztor, Haibo Chen, Zhihai Xu, John Zhuge, Eric Badger,
Kuhu Shukla, Larry Mccay, Rakesh Radhakrishnan, Naganarasimha Garla,
Varun Saxena

1 binding +0, from:
Steve Loughran

and no -1s.

So I am glad to announce that the vote of 2.8.0 RC3 passes.

Thanks everyone listed above who tried the release candidate and vote.
Also, kudos to all who ever help with 2.8.0 release effort in all kinds of ways.
Without working together in community, we cannot afford so many issues get found
in RC stage (as a minor release with debit of ~2 years commits) and most of them
get fixed so quickly.

I'll push the release bits and send out an announcement for 2.8.0 soon.


Thanks,

Junping​



From: Karthik Kambatla <ka...@cloudera.com>
Sent: Wednesday, March 22, 2017 2:10 PM
To: varunsax...@apache.org
Cc: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

+1 (binding)

* Built from source
* Started a pseudo-distributed cluster with fairscheduler.
* Ran sample jobs
* Verified WebUI

On Wed, Mar 22, 2017 at 11:56 AM, 
varunsax...@apache.org<mailto:varunsax...@apache.org> 
<varun.saxena.apa...@gmail.com<mailto:varun.saxena.apa...@gmail.com>> wrote:
Thanks Junping for creating the release.

+1 (non-binding)

* Verified signatures.
* Built from source.
* Set up a pseudo-distributed cluster.
* Successfully ran pi and wordcount jobs.
* Navigated the YARN RM and NM UI.

Regards,
Varun Saxena.

On Wed, Mar 22, 2017 at 12:13 AM, Haibo Chen 
<haiboc...@cloudera.com<mailto:haiboc...@cloudera.com>> wrote:

> Thanks Junping for working on the new release!
>
> +1 non-binding
>
> 1) Downloaded the source, verified the checksum
> 2) Built natively from source, and deployed it to a pseudo-distributed
> cluster
> 3) Ran sleep and teragen job and checked both YARN and JHS web UI
> 4) Played with yarn + mapreduce command lines
>
> Best,
> Haibo Chen
>
> On Mon, Mar 20, 2017 at 11:18 AM, Junping Du 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
>
> > ?Thanks for update, John. Then we should be OK with fixing this issue in
> > 2.8.1.
> >
> > Mark the target version of HADOOP-14205 to 2.8.1 instead of 2.8.0 and
> bump
> > up to blocker in case we could miss this in releasing 2.8.1. :)
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> > 
> > From: John Zhuge <jzh...@cloudera.com<mailto:jzh...@cloudera.com>>
> > Sent: Monday, March 20, 2017 10:31 AM
> > To: Junping Du
> > Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
> > hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>;
> > yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>; 
> > mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>
> > Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >
> > Yes, it only affects ADL. There is a workaround of adding these 2
> > properties to core-site.xml:
> >
> >   
> > fs.adl.impl
> > org.apache.hadoop.fs.adl.AdlFileSystem
> >   
> >
> >   
> > fs.AbstractFileSystem.adl.impl
> > org.apache.hadoop.fs.adl.Adl
> >   
> >
> > I have the initial patch ready but hitting these live unit test failures:
> >
> > Failed tests:
> >   TestAdlFileSystemContractLive.runTest:60->FileSystemContract
> BaseTest.testListStatus:257
> > expected:<1> but was:<10>
> >
> > Tests in error:
> >   TestAdlFileContextMainOperationsLive>FileContextMainOperatio
> nsBaseTest.
> > testMkdirsFailsForSubdirectoryOfExistingFile:254 » AccessControl
> >   TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> > testMkdirsFailsForSubdirectoryOfExistingFile:190 » AccessControl
> >
> >
> > Stay tuned...
> >
> > John Zhuge
> > Software Engineer, Cloudera
> >
>

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Thanks all for response with verification work and vote!


Sounds like we are hitting several issues here, although none seems to be 
blockers so far. Given the large commit set - 2000+ commits first landed in 
branch-2 release, we may should follow 2.7.0 practice that to claim this 
release is not for production cluster, just like Vinod's suggestion in previous 
email. We should quickly come up with 2.8.1 release in next 1 or 2 month for 
production deployment.


We will close the vote in next 24 hours. For people who haven't vote, please 
keep on verification work and report any issues if founded - I will check if 
another round of RC is needed based on your findings. Thanks!


Thanks,


Junping



From: Kuhu Shukla <kshu...@yahoo-inc.com>
Sent: Tuesday, March 21, 2017 3:17 PM
Cc: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)


+1 (non-binding)

- Verified signatures.
- Downloaded and built from source tar.gz.
- Deployed a pseudo-distributed cluster on Mac Sierra.
- Ran example Sleep job successfully.
- Deployed latest Apache Tez 0.9 and ran sample Tez orderedwordcount 
successfully.

Thank you Junping and everyone else who worked on getting this release out.

Warm Regards,
Kuhu
On Tuesday, March 21, 2017, 3:42:46 PM CDT, Eric Badger 
<ebad...@yahoo-inc.com.INVALID> wrote:
+1 (non-binding)

- Verified checksums and signatures of all files
- Built from source on MacOS Sierra via JDK 1.8.0 u65
- Deployed single-node cluster
- Successfully ran a few sample jobs

Thanks,

Eric

On Tuesday, March 21, 2017 2:56 PM, John Zhuge <john.zh...@gmail.com> wrote:



+1. Thanks for the great effort, Junping!


  - Verified checksums and signatures of the tarballs
  - Built source code with Java 1.8.0_66-b17 on Mac OS X 10.12.3
  - Built source and native code with Java 1.8.0_111 on Centos 7.2.1511
  - Cloud connectors:
  - s3a: integration tests, basic fs commands
  - adl: live unit tests, basic fs commands. See notes below.
  - Deployed a pseudo cluster, passed the following sanity tests in
  both insecure and SSL mode:
  - HDFS: basic dfs, distcp, ACL commands
  - KMS and HttpFS: basic tests
  - MapReduce wordcount
  - balancer start/stop


Needs the following JIRAs to pass all ADL tests:

  - HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge.
  - HDFS-11132. Allow AccessControlException in contract tests when
  getFileStatus on subdirectory of existing files. Contributed by Vishwajeet
  Dusane
  - HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1
  runtime error. (John Zhuge via lei)


On Mon, Mar 20, 2017 at 10:31 AM, John Zhuge <jzh...@cloudera.com> wrote:

> Yes, it only affects ADL. There is a workaround of adding these 2
> properties to core-site.xml:
>
>  
>fs.adl.impl
>org.apache.hadoop.fs.adl.AdlFileSystem
>  
>
>  
>fs.AbstractFileSystem.adl.impl
>org.apache.hadoop.fs.adl.Adl
>  
>
> I have the initial patch ready but hitting these live unit test failures:
>
> Failed tests:
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testListStatus:257
> expected:<1> but was:<10>
>
> Tests in error:
>
> TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:254
> » AccessControl
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:190
> » AccessControl
>
>
> Stay tuned...
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du <j...@hortonworks.com> wrote:
>
> > Thank you for reporting the issue, John! Does this issue only affect ADL
> > (Azure Data Lake) which is a new feature for 2.8 rather than other
> existing
> > FS? If so, I think we can leave the fix to 2.8.1 to fix given this is
> not a
> > regression and just a new feature get broken.?
> >
> >
> > Thanks,
> >
> >
> > Junping
> > --
> > *From:* John Zhuge <jzh...@cloudera.com>
> > *Sent:* Monday, March 20, 2017 9:07 AM
> > *To:* Junping Du
> > *Cc:* common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> > *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >
> > Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
> > FileSystem for scheme: adl".
> >
> > The issue were caused by backporting HADOOP-13037 to branch-2 and
> earlier.
> > HADOOP-12666 should not be backported, but so

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

?Thanks for update, John. Then we should be OK with fixing this issue in 2.8.1.

Mark the target version of HADOOP-14205 to 2.8.1 instead of 2.8.0 and bump up 
to blocker in case we could miss this in releasing 2.8.1. :)


Thanks,


Junping


From: John Zhuge <jzh...@cloudera.com>
Sent: Monday, March 20, 2017 10:31 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Yes, it only affects ADL. There is a workaround of adding these 2 properties to 
core-site.xml:

  
fs.adl.impl
org.apache.hadoop.fs.adl.AdlFileSystem
  

  
fs.AbstractFileSystem.adl.impl
org.apache.hadoop.fs.adl.Adl
  

I have the initial patch ready but hitting these live unit test failures:

Failed tests:
  
TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.testListStatus:257
 expected:<1> but was:<10>

Tests in error:
  
TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:254
 » AccessControl
  
TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:190
 » AccessControl


Stay tuned...

John Zhuge
Software Engineer, Cloudera

On Mon, Mar 20, 2017 at 10:02 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:

Thank you for reporting the issue, John! Does this issue only affect ADL (Azure 
Data Lake) which is a new feature for 2.8 rather than other existing FS? If so, 
I think we can leave the fix to 2.8.1 to fix given this is not a regression and 
just a new feature get broken.?


Thanks,


Junping


From: John Zhuge <jzh...@cloudera.com<mailto:jzh...@cloudera.com>>
Sent: Monday, March 20, 2017 9:07 AM
To: Junping Du
Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>; 
mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No FileSystem 
for scheme: adl".

The issue were caused by backporting HADOOP-13037 to branch-2 and earlier. 
HADOOP-12666 should not be backported, but some changes are needed: property 
fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.

I am working on a patch.


John Zhuge
Software Engineer, Cloudera

On Fri, Mar 17, 2017 at 2:18 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi all,
 With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Thank you for reporting the issue, John! Does this issue only affect ADL (Azure 
Data Lake) which is a new feature for 2.8 rather than other existing FS? If so, 
I think we can leave the fix to 2.8.1 to fix given this is not a regression and 
just a new feature get broken.?


Thanks,


Junping


From: John Zhuge <jzh...@cloudera.com>
Sent: Monday, March 20, 2017 9:07 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No FileSystem 
for scheme: adl".

The issue were caused by backporting HADOOP-13037 to branch-2 and earlier. 
HADOOP-12666 should not be backported, but some changes are needed: property 
fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.

I am working on a patch.


John Zhuge
Software Engineer, Cloudera

On Fri, Mar 17, 2017 at 2:18 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi all,
 With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping

[VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Hi all,
 With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Junping Du]

Thanks Steve. That's Awesome! I will kick off a new RC soon.
Shall we reopen HDFS-6200 given issues here? Making it in release note of 2.8.0 
could confuse people as it doesn't work in HA deployment.

Thanks,

Junping

From: Steve Loughran
Sent: Thursday, March 16, 2017 7:27 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

> On 16 Mar 2017, at 00:25, Junping Du <j...@hortonworks.com> wrote:
>
> bq. From my read of the poms, hadoop-client depends on hadoop-hdfs-client to 
> pull in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. 
> So I think this affects users of the hadoop-client artifact, which has 
> existed for a long time.
>
> I could miss that. Thanks for reminding! From my quick check: 
> https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client/2.7.3?, it 
> sounds like 669 artifacts from other projects were depending on it.
>
>
> I think we should withdraw the current RC bits. Please stop the verification 
> & vote.
>
> I will kick off another RC immediately when HDFS-11431 get fixed.

is done. hadoop-hdfs without any server-side dependencies is now a 
hadoop-client dependency.

Release notes:

The hadoop-client POM now includes a leaner hdfs-client, stripping out all the 
transitive dependencies on JARs only needed for the Hadoop HDFS daemon itself. 
The specific jars now excluded are: leveldbjni-all, jetty-util, commons-daemon, 
xercesImpl, netty and servlet-api.

This should make downstream projects dependent JARs smaller, and avoid version 
conflict problems with the specific JARs now excluded.

Applications may encounter build problems if they did depend on these JARs, and 
which didn't explicitly include them. There are two fixes for this

* explicitly include the JARs, stating which version of them you want.
* add a dependency on hadoop-hdfs. For Hadoop 2.8+, this will add the missing 
dependencies. For builds against older versions of Hadoop, this will be 
harmless, as hadoop-hdfs and all its dependencies are already pulled in by the 
hadoop-client POM.




-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Junping Du]

bq. From my read of the poms, hadoop-client depends on hadoop-hdfs-client to 
pull in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. 
So I think this affects users of the hadoop-client artifact, which has existed 
for a long time.

I could miss that. Thanks for reminding! From my quick check: 
https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client/2.7.3?, it 
sounds like 669 artifacts from other projects were depending on it.


I think we should withdraw the current RC bits. Please stop the verification & 
vote.

I will kick off another RC immediately when HDFS-11431 get fixed.


Thanks,


Junping



From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Wednesday, March 15, 2017 2:04 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

Hi Junping, inline,


>From my understanding, this issue is related to our previous improvements with 
>separating client and server jars in HDFS-6200. If we use the new "client" jar 
>in NN HA deployment, then we will hit the issue reported.

>From my read of the poms, hadoop-client depends on hadoop-hdfs-client to pull 
>in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. So I 
>think this affects users of the hadoop-client artifact, which has existed for 
>a long time.

Essentially all of our customer deployments run with NN HA, so this would 
affect a lot of users.

I can see two options here:

- Without any change in 2.8.0, if user hit the issue when they deploy HA 
cluster by using new client jar, adding back hdfs jar just like how things work 
previously

- Make the change now in 2.8.0, either moving ConfiguredFailoverProxyProvider 
to client jar or adding dependency between client jar and server jar. There 
must be some arguments there on which way to fix is better especially 
ConfiguredFailoverProxyProvider still has some sever side dependencies.


I would prefer the first option, given:

- The issue fixing time is unpredictable as there are still discussion on how 
to fix this issue. Our 2.8.0 release shouldn't be an endless journey which has 
been deferred several times for more serious issue.

Looks like we have a patch being actively revved and reviewed to fix this by 
making hadoop-hdfs-client depend on hadoop-hdfs. Thanks to Steven and Steve for 
working on this.

Steve proposed doing a proper split in a later JIRA.

- We have workaround for this improvement, no regression happens due to this 
issue. People can still use hdfs jar in old way. The worst case is improvement 
for HDFS doesn't work in some cases - that shouldn't block the whole release.

Based on the above, I think there is a regression for users of the 
hadoop-client artifact.

If it actually only affects users of hadoop-hdfs-client, then I agree we can 
document it as a Known Issue and fix it later.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Junping Du]

Thanks Andrew for reporting the issue. This JIRA is out of my radar as it? 
didn't specify any target version before.


>From my understanding, this issue is related to our previous improvements with 
>separating client and server jars in HDFS-6200. If we use the new "client" jar 
>in NN HA deployment, then we will hit the issue reported.


I can see two options here:

- Without any change in 2.8.0, if user hit the issue when they deploy HA 
cluster by using new client jar, adding back hdfs jar just like how things work 
previously

- Make the change now in 2.8.0, either moving ConfiguredFailoverProxyProvider 
to client jar or adding dependency between client jar and server jar. There 
must be some arguments there on which way to fix is better especially 
ConfiguredFailoverProxyProvider still has some sever side dependencies.


I would prefer the first option, given:

- The issue fixing time is unpredictable as there are still discussion on how 
to fix this issue. Our 2.8.0 release shouldn't be an endless journey which has 
been deferred several times for more serious issue.

- We have workaround for this improvement, no regression happens due to this 
issue. People can still use hdfs jar in old way. The worst case is improvement 
for HDFS doesn't work in some cases - that shouldn't block the whole release.


I think we should let vote keep going unless someone have more concerns which I 
could miss.



Thanks,


Junping



From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Tuesday, March 14, 2017 2:50 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

Hi Junping,

Noticed this possible blocker float by my inbox today. It had an affects but no 
target version set:

https://issues.apache.org/jira/browse/HDFS-11431

Thoughts? Seems like the hadoop-hdfs-client artifact doesn't work right now.

Best,
Andrew


On Tue, Mar 14, 2017 at 1:41 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi all,
 With several important fixes get merged last week, I've created a new 
release candidate (RC2) for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,919 fixes, improvements, and new features. 
Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  Please note that RC0 and RC1 are not voted public because significant 
issues are found just after RC tag getting published.

  The RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC2

  The RC tag in git is: release-2.8.0-RC2

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1056

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/20/2017 PDT time.

Thanks,

Junping

[jira] [Created] (YARN-6336) Jenkins report YARN new UI build failure

[Junping Du (JIRA)]

Junping Du created YARN-6336:


 Summary: Jenkins report YARN new UI build failure 
 Key: YARN-6336
 URL: https://issues.apache.org/jira/browse/YARN-6336
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Junping Du
Priority: Blocker


In Jenkins report of YARN-6313 
(https://builds.apache.org/job/PreCommit-YARN-Build/15260/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt),
 we found following build failure due to YARN new UI:
{noformat}
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/ember-cli-htmlbars/node_modules/broccoli-persistent-filter/node_modules/async-disk-cache/node_modules/username/index.js:2
const os = require('os');
^
Use of const in strict mode.
SyntaxError: Use of const in strict mode.
at Module._compile (module.js:439:25)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. 
(/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/ember-cli-htmlbars/node_modules/broccoli-persistent-filter/node_modules/async-disk-cache/index.js:24:16)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
DEPRECATION: Node v0.10.25 is no longer supported by Ember CLI. Please update 
to a more recent version of Node
undefined
version: 1.13.15
Could not find watchman, falling back to NodeWatcher for file system events.
Visit http://www.ember-cli.com/user-guide/#watchman for more info.
Building[INFO] 

{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Junping Du]

Hi all,
 With several important fixes get merged last week, I've created a new 
release candidate (RC2) for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,919 fixes, improvements, and new features. 
Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  Please note that RC0 and RC1 are not voted public because significant 
issues are found just after RC tag getting published.

  The RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC2

  The RC tag in git is: release-2.8.0-RC2

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1056

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/20/2017 PDT time.

Thanks,

Junping

[jira] [Created] (YARN-6294) ATS client should better handle Socket closed case

[Junping Du (JIRA)]

Junping Du created YARN-6294:


 Summary: ATS client should better handle Socket closed case
 Key: YARN-6294
 URL: https://issues.apache.org/jira/browse/YARN-6294
 Project: Hadoop YARN
  Issue Type: Bug
  Components: timelineclient
Reporter: Sumana Sathish
Assignee: Li Lu


Exception stack:
{noformat}
17/02/06 07:11:30 INFO distributedshell.ApplicationMaster: Container completed 
successfully., containerId=container_1486362713048_0037_01_02
17/02/06 07:11:30 ERROR distributedshell.ApplicationMaster: Error in 
RMCallbackHandler: 
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: 
Socket closed
at 
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter$1.run(TimelineClientImpl.java:236)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:185)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter.handle(TimelineClientImpl.java:248)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at 
com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPostingObject(TimelineWriter.java:154)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineWriter$1.run(TimelineWriter.java:115)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineWriter$1.run(TimelineWriter.java:112)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1833)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPosting(TimelineWriter.java:112)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineWriter.putEntities(TimelineWriter.java:92)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:346)
at 
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.publishContainerEndEvent(ApplicationMaster.java:1145)
at 
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.access$400(ApplicationMaster.java:169)
at 
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$RMCallbackHandler.onContainersCompleted(ApplicationMaster.java:779)
at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:296)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at 
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at 
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
at 
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 20 more
Exception in thread "AMRM Callback Handler Thread" 
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Yes. I did maven deploy in root directory before close the staging repository. 
If this is the only suspect, I can drop the repository and do mvn deploy again.


Thanks,


Junping


From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Friday, January 20, 2017 10:48 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

You can check the error message by clicking on it, a bunch like this:

Missing Signature: 
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.8.0/hadoop-mapreduce-client-jobclient-2.8.0-tests.jar.asc'
 does not exist for 'hadoop-mapreduce-client-jobclient-2.8.0-tests.jar'.

Did you maven deploy all the signature files?

On Fri, Jan 20, 2017 at 2:04 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi,
I have successfully built the release bit on branch-2.8.0 by following: 
https://wiki.apache.org/hadoop/HowToRelease step by step. However, when try to 
close the "Staging Repositories" at Nexus page 
(https://repository.apache.org/#stagingRepositories), I found our repository - 
orgapachehadoop-1051 cannot be closed due to some signature validation failed 
and one Apache rule failed. I never met this problem before in previous 
releasing 2.6.3 and 2.6.4. Is this related to our recent changes on release 
process/tools (docker based)? Any ideas or thoughts on how to fix the problem?

Thanks,

Junping

____________
From: Junping Du <j...@hortonworks.com<mailto:j...@hortonworks.com>>
Sent: Thursday, January 19, 2017 6:46 PM
To: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>; 
yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>
Cc: Varun Vasudev
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

According to Varun's offline email, the security fixes has landed on branch-2, 
2.8 and 2.8.0 branch.
I was kicking off a new RC build (RC1), and will publish it for vote soon. In 
the mean time, please mark fix version as 2.8.1 for any new commits landed on 
branch-2.8, and don't commit anything to branch-2.8.0 at this moment. Thanks!

Cheers,

Junping


From: Junping Du <j...@hortonworks.com<mailto:j...@hortonworks.com>>
Sent: Wednesday, January 18, 2017 3:26 PM
To: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>; 
yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>
Cc: Varun Vasudev
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi folks,
 In the passed one or two weeks, we found some new blockers get coming on 
branch-2.8, like: YARN-6068 (log aggregation get stuck when NM restart with 
work preserving enabled) and YARN-6072 (RM unable to start in secure mode). 
Both of them are fixed now (YARN-6072 is fixed by Ajith and I fixed YARN-6068), 
and I was starting the RC build process since early this week. As we have 
significant build tools/process change (docker based) since 2.8 comparing with 
2.6/2.7, it takes me a while to get familiar with it and finally get a 
successful build on 2.8.0-RC0 last night.
 I already push RC0 tag into public which is prerequisite step before RC 
voting. However,  in the mean while, I was pinged by Varun Vasudev that there 
are a known vulnerability issues on container_executor get identified and 
discussed in hadoop security email threads - looks like YARN-5704 fixed part of 
it, but left part - the privilege escalation via /proc/self/environ is not 
fixed yet. So most likely, I have to withdraw our 2.8.0 RC0 although I haven't 
announced it public for vote yet. I will wait this issue get fixed to prepare a 
new release candidate. As RC0 tag cannot be reverted after push into apache, 
our next release candidate will start from RC1.
As I mentioned in early email, 2.8.0 is a very big release (2300+ commits 
since 2.7.3) and I am glad that we are almost there. Thanks everyone for being 
patient and contributing our release work. Please let me know if you have more 
comments or suggestions.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com<mailto:j...@hortonworks.com>>
Sent: Wednesday, January 04, 2017 1:41 AM
To: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>; 
yarn-dev@hadoop.apache.o

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Hi,
I have successfully built the release bit on branch-2.8.0 by following: 
https://wiki.apache.org/hadoop/HowToRelease step by step. However, when try to 
close the "Staging Repositories" at Nexus page 
(https://repository.apache.org/#stagingRepositories), I found our repository - 
orgapachehadoop-1051 cannot be closed due to some signature validation failed 
and one Apache rule failed. I never met this problem before in previous 
releasing 2.6.3 and 2.6.4. Is this related to our recent changes on release 
process/tools (docker based)? Any ideas or thoughts on how to fix the problem?

Thanks,

Junping

____
From: Junping Du <j...@hortonworks.com>
Sent: Thursday, January 19, 2017 6:46 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Cc: Varun Vasudev
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

According to Varun's offline email, the security fixes has landed on branch-2, 
2.8 and 2.8.0 branch.
I was kicking off a new RC build (RC1), and will publish it for vote soon. In 
the mean time, please mark fix version as 2.8.1 for any new commits landed on 
branch-2.8, and don't commit anything to branch-2.8.0 at this moment. Thanks!

Cheers,

Junping

________
From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, January 18, 2017 3:26 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Cc: Varun Vasudev
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi folks,
 In the passed one or two weeks, we found some new blockers get coming on 
branch-2.8, like: YARN-6068 (log aggregation get stuck when NM restart with 
work preserving enabled) and YARN-6072 (RM unable to start in secure mode). 
Both of them are fixed now (YARN-6072 is fixed by Ajith and I fixed YARN-6068), 
and I was starting the RC build process since early this week. As we have 
significant build tools/process change (docker based) since 2.8 comparing with 
2.6/2.7, it takes me a while to get familiar with it and finally get a 
successful build on 2.8.0-RC0 last night.
 I already push RC0 tag into public which is prerequisite step before RC 
voting. However,  in the mean while, I was pinged by Varun Vasudev that there 
are a known vulnerability issues on container_executor get identified and 
discussed in hadoop security email threads - looks like YARN-5704 fixed part of 
it, but left part - the privilege escalation via /proc/self/environ is not 
fixed yet. So most likely, I have to withdraw our 2.8.0 RC0 although I haven't 
announced it public for vote yet. I will wait this issue get fixed to prepare a 
new release candidate. As RC0 tag cannot be reverted after push into apache, 
our next release candidate will start from RC1.
As I mentioned in early email, 2.8.0 is a very big release (2300+ commits 
since 2.7.3) and I am glad that we are almost there. Thanks everyone for being 
patient and contributing our release work. Please let me know if you have more 
comments or suggestions.

Thanks,

Junping
________
From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, January 04, 2017 1:41 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi all Hadoopers,
 I just commit YARN-3866 which is the last blocker for branch-2.8, so since 
tomorrow I will kick off process to prepare the first RC for our 2.8.0 release. 
This release will include at least 2374 commits that never landed before in 
previous 2.x releases. Please check https://s.apache.org/RW5k for details. 
There are also 352 fixes marked in 2.7.1, 2.7.2 and 2.7.3 but not marked with 
2.8 (https://s.apache.org/RKti) that I need to double check all are landed in 
branch-2.8 and fix the versions before kicking off the first RC.
 Our progress is slightly behind my estimation weeks ago. However, 
considering we just go through holidays and Jenkins cleanup issues linger on 
for several projects (YARN, etc.), our achievement here is still great! Thanks 
everyone for keep calm and carry on the help for the release. Kudos to Wangda, 
Jian, Akira, Jason, Sangjin, Karthik and all for pushing hard on blockers 
during this time. Also, special thanks to Vinod and Andrew for sharing 
knowledge and practice (include scripts for auto version check) for releasing 
effort.
 Will try to prepare RC0 of 2.8 release for vote within this week. Stay 
tuned!

Thanks,

Junping

____________
From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, December 07, 2016 11:31 AM
To: Akira Ajisaka; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

According to Varun's offline email, the security fixes has landed on branch-2, 
2.8 and 2.8.0 branch. 
I was kicking off a new RC build (RC1), and will publish it for vote soon. In 
the mean time, please mark fix version as 2.8.1 for any new commits landed on 
branch-2.8, and don't commit anything to branch-2.8.0 at this moment. Thanks!

Cheers,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, January 18, 2017 3:26 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Cc: Varun Vasudev
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi folks,
 In the passed one or two weeks, we found some new blockers get coming on 
branch-2.8, like: YARN-6068 (log aggregation get stuck when NM restart with 
work preserving enabled) and YARN-6072 (RM unable to start in secure mode). 
Both of them are fixed now (YARN-6072 is fixed by Ajith and I fixed YARN-6068), 
and I was starting the RC build process since early this week. As we have 
significant build tools/process change (docker based) since 2.8 comparing with 
2.6/2.7, it takes me a while to get familiar with it and finally get a 
successful build on 2.8.0-RC0 last night.
 I already push RC0 tag into public which is prerequisite step before RC 
voting. However,  in the mean while, I was pinged by Varun Vasudev that there 
are a known vulnerability issues on container_executor get identified and 
discussed in hadoop security email threads - looks like YARN-5704 fixed part of 
it, but left part - the privilege escalation via /proc/self/environ is not 
fixed yet. So most likely, I have to withdraw our 2.8.0 RC0 although I haven't 
announced it public for vote yet. I will wait this issue get fixed to prepare a 
new release candidate. As RC0 tag cannot be reverted after push into apache, 
our next release candidate will start from RC1.
As I mentioned in early email, 2.8.0 is a very big release (2300+ commits 
since 2.7.3) and I am glad that we are almost there. Thanks everyone for being 
patient and contributing our release work. Please let me know if you have more 
comments or suggestions.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, January 04, 2017 1:41 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi all Hadoopers,
 I just commit YARN-3866 which is the last blocker for branch-2.8, so since 
tomorrow I will kick off process to prepare the first RC for our 2.8.0 release. 
This release will include at least 2374 commits that never landed before in 
previous 2.x releases. Please check https://s.apache.org/RW5k for details. 
There are also 352 fixes marked in 2.7.1, 2.7.2 and 2.7.3 but not marked with 
2.8 (https://s.apache.org/RKti) that I need to double check all are landed in 
branch-2.8 and fix the versions before kicking off the first RC.
 Our progress is slightly behind my estimation weeks ago. However, 
considering we just go through holidays and Jenkins cleanup issues linger on 
for several projects (YARN, etc.), our achievement here is still great! Thanks 
everyone for keep calm and carry on the help for the release. Kudos to Wangda, 
Jian, Akira, Jason, Sangjin, Karthik and all for pushing hard on blockers 
during this time. Also, special thanks to Vinod and Andrew for sharing 
knowledge and practice (include scripts for auto version check) for releasing 
effort.
 Will try to prepare RC0 of 2.8 release for vote within this week. Stay 
tuned!

Thanks,

Junping

____
From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, December 07, 2016 11:31 AM
To: Akira Ajisaka; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for 
several runs before last weekend, but it get failed for latest several runs, 
probably affected by recently Jenkins down.
However, from my recent manual kick off, it seems to be good again: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I will 
keep an eye for today's nightly run to see if it back to normal.


Thanks,

Junping


From: Akira Ajisaka <ajisa...@apache.org>
Sent: Wednesday, December 07, 2016 12:12 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Junping and Andrew!

HADOOP-2.8-JACC is not working well, so I manually kicked a job to
compare 2.8 with 2.7.3.

https://builds.

Re: [VOTE] Release cadence and EOL

[Junping Du]

+1 on Sangjin's proposal - 
"A minor release line is end-of-lifed 2 years after it is released or there
are 2 newer minor releases, whichever is sooner. The community reserves the
right to extend or shorten the life of a release line if there is a good
reason to do so."

I also noticed Karthik bring up some new proposals - some of them looks 
interesting to me and I have some ideas as well. Karthik, can you bring it out 
in a separated discussion threads so that we can discuss from there?

About Chris Trezzo's question about definition of EOL of hadoop release, I 
think potentially changes could be: 
1. For users of Apache hadoop, they would expect to upgrade to a new 
minor/major releases after EOL of their current release because there is no 
guarantee of new maintenance release.

2. For release effort, apache law claim that committer can volunteer RM for any 
release. With this release EOL proposal passes and written into hadoop bylaw, 
anyone want to call for a release which is EOL then she/he have to provide a 
good reason to community and get voted before to start release effort. We don't 
want to waste community time/resource to verify/vote a narrow interested 
release.

3. About committer's responsibility, I think the bottom line is committer 
should commit patch contributor's target release and her/his own interest 
release which I conservatively agree with Allen's point that this vote doesn't 
change anything. But if a committer want to take care more interest from the 
whole community like most committers are doing today, he/she should understand 
which branches can benefit more people and could skip some EOL release branches 
for backport effort.

About major release EOL, this could be more complicated and I think we should 
discuss separately.

Thanks,

Junping

From: Allen Wittenauer 
Sent: Wednesday, January 18, 2017 3:30 PM
To: Chris Trezzo
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release cadence and EOL

> On Jan 18, 2017, at 11:21 AM, Chris Trezzo  wrote:
>
> Thanks Sangjin for pushing this forward! I have a few questions:

These are great questions, because I know I'm not seeing a whole lot of 
substance in this vote.  The way to EOL software in the open source universe is 
with new releases and aging it out.  If someone wants to be a RE for a new 
branch-1 release, more power to them.  As volunteers to the ASF, we're not on 
the hook to provide much actual support.  This feels more like a vendor play 
than a community one.  But if the PMC want to vote on it, whatever.  It won't 
be first bylaw that doesn't really mean much.

> 1. What is the definition of end-of-life for a release in the hadoop
> project? My current understanding is as follows: When a release line
> reaches end-of-life, there are no more planned releases for that line.
> Committers are no longer responsible for back-porting bug fixes to the line
> (including fixed security vulnerabilities) and it is essentially
> unmaintained.

Just a point of clarification.  There is no policy that says that 
committers must back port.  It's up to the individual committers to push a 
change onto any particular branch. Therefore, this vote doesn't really change 
anything in terms of committer responsibilities here.

> 2. How do major releases affect the end-of-life proposal? For example, how
> does a new minor release in the next major release affect the end-of-life
> of minor releases in a previous major release? Is it possible to have a
> maintained 2.x release if there is a 3.3 release?

I'm looking forward to seeing this answer too, given that 2.7.0 is 
probably past the 2 year mark, 2.8.0 has seemingly been in a holding pattern 
for over a year, and the next 3.0.0 alpha should be RSN

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Hi folks,
 In the passed one or two weeks, we found some new blockers get coming on 
branch-2.8, like: YARN-6068 (log aggregation get stuck when NM restart with 
work preserving enabled) and YARN-6072 (RM unable to start in secure mode). 
Both of them are fixed now (YARN-6072 is fixed by Ajith and I fixed YARN-6068), 
and I was starting the RC build process since early this week. As we have 
significant build tools/process change (docker based) since 2.8 comparing with 
2.6/2.7, it takes me a while to get familiar with it and finally get a 
successful build on 2.8.0-RC0 last night. 
 I already push RC0 tag into public which is prerequisite step before RC 
voting. However,  in the mean while, I was pinged by Varun Vasudev that there 
are a known vulnerability issues on container_executor get identified and 
discussed in hadoop security email threads - looks like YARN-5704 fixed part of 
it, but left part - the privilege escalation via /proc/self/environ is not 
fixed yet. So most likely, I have to withdraw our 2.8.0 RC0 although I haven't 
announced it public for vote yet. I will wait this issue get fixed to prepare a 
new release candidate. As RC0 tag cannot be reverted after push into apache, 
our next release candidate will start from RC1.
As I mentioned in early email, 2.8.0 is a very big release (2300+ commits 
since 2.7.3) and I am glad that we are almost there. Thanks everyone for being 
patient and contributing our release work. Please let me know if you have more 
comments or suggestions.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, January 04, 2017 1:41 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi all Hadoopers,
 I just commit YARN-3866 which is the last blocker for branch-2.8, so since 
tomorrow I will kick off process to prepare the first RC for our 2.8.0 release. 
This release will include at least 2374 commits that never landed before in 
previous 2.x releases. Please check https://s.apache.org/RW5k for details. 
There are also 352 fixes marked in 2.7.1, 2.7.2 and 2.7.3 but not marked with 
2.8 (https://s.apache.org/RKti) that I need to double check all are landed in 
branch-2.8 and fix the versions before kicking off the first RC.
 Our progress is slightly behind my estimation weeks ago. However, 
considering we just go through holidays and Jenkins cleanup issues linger on 
for several projects (YARN, etc.), our achievement here is still great! Thanks 
everyone for keep calm and carry on the help for the release. Kudos to Wangda, 
Jian, Akira, Jason, Sangjin, Karthik and all for pushing hard on blockers 
during this time. Also, special thanks to Vinod and Andrew for sharing 
knowledge and practice (include scripts for auto version check) for releasing 
effort.
 Will try to prepare RC0 of 2.8 release for vote within this week. Stay 
tuned!

Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, December 07, 2016 11:31 AM
To: Akira Ajisaka; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for 
several runs before last weekend, but it get failed for latest several runs, 
probably affected by recently Jenkins down.
However, from my recent manual kick off, it seems to be good again: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I will 
keep an eye for today's nightly run to see if it back to normal.


Thanks,

Junping


From: Akira Ajisaka <ajisa...@apache.org>
Sent: Wednesday, December 07, 2016 12:12 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Junping and Andrew!

HADOOP-2.8-JACC is not working well, so I manually kicked a job to
compare 2.8 with 2.7.3.

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/24/artifact/target/compat-check/report.html

Regards,
Akira

On 2016/12/02 2:08, Junping Du wrote:
> Thanks Andrew! That's also a nice suggestion. I already create a similar job: 
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/ for 2.8 
> and kick off several runs manually. Will monitor incompatible status from 
> there.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Andrew Wang <andrew.w...@cloudera.com>
> Sent: Wednesday, November 30, 2016 4:18 PM
> To: Junping Du
> Cc: Sangjin Lee; common-...@hadoop.apache.org; hdfs-...@hadoop

[jira] [Resolved] (YARN-6079) simple spelling errors in yarn test code

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du resolved YARN-6079.
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   2.9.0

> simple spelling errors in yarn test code
> 
>
> Key: YARN-6079
> URL: https://issues.apache.org/jira/browse/YARN-6079
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Grant Sohn
>Assignee: vijay
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-6079.001.patch
>
>
> charactor -> character
> hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestCommonNodeLabelsManager.java:
> Assert.assertTrue("invalid label charactor should not add to repo", 
> caught);
> expteced -> expected
> hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java:
>   Assert.fail("Exception is not expteced.");
> Exepected -> Expected
> hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java:
> "Exepected AbsoluteUsedCapacity > 0.95, got: "
> expteced -> expected
> hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java:
>   Assert.fail("Exception is not expteced.");
> macthing -> matching
> hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java:
> assertEquals("Expected no macthing requests.", matches.size(), 0);
> propogated -> propagated
> hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeHealthService.java:
> Assert.assertTrue("Node script time out message not propogated",
> protential -> potential
> hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/BasePBImplRecordsTest.java:
> LOG.info(String.format("Exclude protential property: %s\n", 
> gsp.propertyName));
> recevied -> received
> hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java:
> throw new Exception("Unexpected resource recevied.");
> shouldnt -> shouldn't
> hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServiceAppsNodelabel.java:
>   fail("resourceInfo object shouldnt be available for finished apps");
> Transistion -> Transition
> hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHA.java:
>   Assert.fail("Transistion to Active should have failed for 
> refreshAll()");
> Unhelathy -> Unhealthy
> hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java:
> Assert.assertEquals("Unhelathy Nodes", initialUnHealthy,



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6071) Fix incompatible API change on AM-RM protocol due to YARN-3866 (trunk only)

[Junping Du (JIRA)]

Junping Du created YARN-6071:


 Summary: Fix incompatible API change on AM-RM protocol due to 
YARN-3866 (trunk only)
 Key: YARN-6071
 URL: https://issues.apache.org/jira/browse/YARN-6071
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Junping Du
Assignee: Wangda Tan
Priority: Blocker


In YARN-3866, we have addendum patch to fix incompatible API change on branch-2 
and branch-2.8. For trunk, we need a similar fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6068) Log aggregation get failed when NM restart even with recovery

[Junping Du (JIRA)]

Junping Du created YARN-6068:


 Summary: Log aggregation get failed when NM restart even with 
recovery
 Key: YARN-6068
 URL: https://issues.apache.org/jira/browse/YARN-6068
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Junping Du
Assignee: Junping Du
Priority: Critical


The exception log is as following:
{noformat}
2017-01-05 19:16:36,352 INFO  logaggregation.AppLogAggregatorImpl 
(AppLogAggregatorImpl.java:abortLogAggregation(527)) - Aborting log aggregation 
for application_1483640789847_0001
2017-01-05 19:16:36,352 WARN  logaggregation.AppLogAggregatorImpl 
(AppLogAggregatorImpl.java:run(399)) - Aggregation did not complete for 
application application_1483640789847_0001
2017-01-05 19:16:36,353 WARN  application.ApplicationImpl 
(ApplicationImpl.java:handle(461)) - Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
APPLICATION_LOG_HANDLING_FAILED at RUNNING
at 
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at 
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at 
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:459)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:64)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1084)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1076)
at 
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
at 
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:745)
2017-01-05 19:16:36,355 INFO  application.ApplicationImpl 
(ApplicationImpl.java:handle(464)) - Application application_1483640789847_0001 
transitioned from RUNNING to null
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Hi all Hadoopers,
 I just commit YARN-3866 which is the last blocker for branch-2.8, so since 
tomorrow I will kick off process to prepare the first RC for our 2.8.0 release. 
This release will include at least 2374 commits that never landed before in 
previous 2.x releases. Please check https://s.apache.org/RW5k for details. 
There are also 352 fixes marked in 2.7.1, 2.7.2 and 2.7.3 but not marked with 
2.8 (https://s.apache.org/RKti) that I need to double check all are landed in 
branch-2.8 and fix the versions before kicking off the first RC.
 Our progress is slightly behind my estimation weeks ago. However, 
considering we just go through holidays and Jenkins cleanup issues linger on 
for several projects (YARN, etc.), our achievement here is still great! Thanks 
everyone for keep calm and carry on the help for the release. Kudos to Wangda, 
Jian, Akira, Jason, Sangjin, Karthik and all for pushing hard on blockers 
during this time. Also, special thanks to Vinod and Andrew for sharing 
knowledge and practice (include scripts for auto version check) for releasing 
effort.
 Will try to prepare RC0 of 2.8 release for vote within this week. Stay 
tuned!

Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, December 07, 2016 11:31 AM
To: Akira Ajisaka; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for 
several runs before last weekend, but it get failed for latest several runs, 
probably affected by recently Jenkins down.
However, from my recent manual kick off, it seems to be good again: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I will 
keep an eye for today's nightly run to see if it back to normal.


Thanks,

Junping


From: Akira Ajisaka <ajisa...@apache.org>
Sent: Wednesday, December 07, 2016 12:12 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Junping and Andrew!

HADOOP-2.8-JACC is not working well, so I manually kicked a job to
compare 2.8 with 2.7.3.

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/24/artifact/target/compat-check/report.html

Regards,
Akira

On 2016/12/02 2:08, Junping Du wrote:
> Thanks Andrew! That's also a nice suggestion. I already create a similar job: 
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/ for 2.8 
> and kick off several runs manually. Will monitor incompatible status from 
> there.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Andrew Wang <andrew.w...@cloudera.com>
> Sent: Wednesday, November 30, 2016 4:18 PM
> To: Junping Du
> Cc: Sangjin Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Vinod 
> Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> I recommend giving the JACC report another look. I set up a parameterized 
> jenkins job which you can trigger manually for branch-2.8:
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/
>
> On Wed, Nov 30, 2016 at 4:06 PM, Junping Du 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
> Hi Sangjin and all,
>
>  That sounds good. If anyone have priority fix to backport , please 
> nominate it by following this email thread or ping me on JIRA directly. And I 
> agree that we should keep in mind that our bar for 2.8 release should be high 
> now as we want to move quickly on this release. Non-critical fixes can wait 
> for next one.
>
>   Any other comments and suggestions?
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: sjl...@gmail.com<mailto:sjl...@gmail.com> 
> <sjl...@gmail.com<mailto:sjl...@gmail.com>> on behalf of Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org>>
> Sent: Wednesday, November 30, 2016 3:24 PM
> To: Junping Du
> Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
> hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
> mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>; 
> yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>; Vinod 
> Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com<mailto:sj...@twitter.com>
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> T

Re: YARN Jenkins Build get consistent failed.

[Junping Du]

Thanks Ted. I noticed that Jenkins test for YARN is back to work now. Resolving 
the infra ticket.


Thanks,


Junping



From: Ted Yu <yuzhih...@gmail.com>
Sent: Wednesday, December 21, 2016 12:25 PM
To: Junping Du
Cc: yarn-dev@hadoop.apache.org; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: YARN Jenkins Build get consistent failed.

Precommit build #14423 has completed.

The exclusion (H5 and H6) has been done.

See the other thread started by Sangjin.

On Wed, Dec 21, 2016 at 11:59 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi hadoop folks,

   I noticed that our recent YARN jenkins tests are consistently failed 
(https://builds.apache.org/job/PreCommit-YARN-Build) due to test environment 
issues below.

   I already filed blocker issue 
https://issues.apache.org/jira/browse/INFRA-13141 to our INFRA team yesterday 
but haven't get any response yet. All commit work on YARN project are fully 
blocked. Anyone have ideas on how to move things forward?

btw, Jenkins tests for hadoop/hdfs/mapreduce seems to be OK.


FATAL: Command "git clean -fdx" returned status code 1:
stdout:
stderr: warning: failed to remove 
hadoop-common-project/hadoop-common/target/test/data/3

hudson.plugins.git.GitException<http://stacktrace.jenkins-ci.org/search?query=hudson.plugins.git.GitException>:
 Command "git clean -fdx" returned status code 1:
stdout:
stderr: warning: failed to remove 
hadoop-common-project/hadoop-common/target/test/data/3

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1699)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1695)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1317)




Thanks,


Junping

YARN Jenkins Build get consistent failed.

[Junping Du]

Hi hadoop folks,

   I noticed that our recent YARN jenkins tests are consistently failed 
(https://builds.apache.org/job/PreCommit-YARN-Build) due to test environment 
issues below.

   I already filed blocker issue 
https://issues.apache.org/jira/browse/INFRA-13141 to our INFRA team yesterday 
but haven't get any response yet. All commit work on YARN project are fully 
blocked. Anyone have ideas on how to move things forward?

btw, Jenkins tests for hadoop/hdfs/mapreduce seems to be OK.


FATAL: Command "git clean -fdx" returned status code 1:
stdout:
stderr: warning: failed to remove 
hadoop-common-project/hadoop-common/target/test/data/3

hudson.plugins.git.GitException:
 Command "git clean -fdx" returned status code 1:
stdout:
stderr: warning: failed to remove 
hadoop-common-project/hadoop-common/target/test/data/3

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1699)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1695)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1317)




Thanks,


Junping

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for 
several runs before last weekend, but it get failed for latest several runs, 
probably affected by recently Jenkins down. 
However, from my recent manual kick off, it seems to be good again: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I will 
keep an eye for today's nightly run to see if it back to normal.


Thanks,

Junping


From: Akira Ajisaka <ajisa...@apache.org>
Sent: Wednesday, December 07, 2016 12:12 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Junping and Andrew!

HADOOP-2.8-JACC is not working well, so I manually kicked a job to
compare 2.8 with 2.7.3.

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/24/artifact/target/compat-check/report.html

Regards,
Akira

On 2016/12/02 2:08, Junping Du wrote:
> Thanks Andrew! That's also a nice suggestion. I already create a similar job: 
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/ for 2.8 
> and kick off several runs manually. Will monitor incompatible status from 
> there.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Andrew Wang <andrew.w...@cloudera.com>
> Sent: Wednesday, November 30, 2016 4:18 PM
> To: Junping Du
> Cc: Sangjin Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Vinod 
> Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> I recommend giving the JACC report another look. I set up a parameterized 
> jenkins job which you can trigger manually for branch-2.8:
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/
>
> On Wed, Nov 30, 2016 at 4:06 PM, Junping Du 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
> Hi Sangjin and all,
>
>  That sounds good. If anyone have priority fix to backport , please 
> nominate it by following this email thread or ping me on JIRA directly. And I 
> agree that we should keep in mind that our bar for 2.8 release should be high 
> now as we want to move quickly on this release. Non-critical fixes can wait 
> for next one.
>
>   Any other comments and suggestions?
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: sjl...@gmail.com<mailto:sjl...@gmail.com> 
> <sjl...@gmail.com<mailto:sjl...@gmail.com>> on behalf of Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org>>
> Sent: Wednesday, November 30, 2016 3:24 PM
> To: Junping Du
> Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
> hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
> mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>; 
> yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>; Vinod 
> Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com<mailto:sj...@twitter.com>
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> Thanks for picking this up Junping!
>
> I heard on email threads and offline discussions that there may be interests 
> in porting some fixes from branch-2 to branch-2.8. I still feel that the bar 
> should be pretty high to do that, but it might be good to have folks suggest 
> some critical fixes that need to be released sooner. Thoughts?
>
> On Tue, Nov 29, 2016 at 8:50 PM, Junping Du 
> <j...@hortonworks.com<mailto:j...@hortonworks.com><mailto:j...@hortonworks.com<mailto:j...@hortonworks.com>>>
>  wrote:
> Hi folks,
>  Per asked by Vinod offline, I would like to continue the effort to push 
> 2.8 release out in next several weeks. The plan is as following:
>  - We only have 5 blockers and 3 critical issues 
> (https://s.apache.org/6kwx) so far, and most of them are in good progress 
> now. I put them all on apache ciwiki ( 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release) and 
> will do daily monitor and update on these issues in next couple of weeks.
>  - When all blocker/critical issues are gone (target EOD is Dec. 10), I 
> will push out all major/minor issues that is open. So if you are working on 
> some major issues and target for release in 2.8, please keep track of our 
> release status and your work progress.
>  - I will create RC immediately once our 2.8 target issues are clean 
> (especially for blocker and critical issues) for voting process.
>
>  I hope we

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Thanks Andrew! That's also a nice suggestion. I already create a similar job: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/ for 2.8 and 
kick off several runs manually. Will monitor incompatible status from there.


Thanks,


Junping



From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Wednesday, November 30, 2016 4:18 PM
To: Junping Du
Cc: Sangjin Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Vinod Vavilapalli; 
Jian He; Wangda Tan; sj...@twitter.com
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

I recommend giving the JACC report another look. I set up a parameterized 
jenkins job which you can trigger manually for branch-2.8:

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/

On Wed, Nov 30, 2016 at 4:06 PM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi Sangjin and all,

 That sounds good. If anyone have priority fix to backport , please 
nominate it by following this email thread or ping me on JIRA directly. And I 
agree that we should keep in mind that our bar for 2.8 release should be high 
now as we want to move quickly on this release. Non-critical fixes can wait for 
next one.

  Any other comments and suggestions?


Thanks,


Junping



From: sjl...@gmail.com<mailto:sjl...@gmail.com> 
<sjl...@gmail.com<mailto:sjl...@gmail.com>> on behalf of Sangjin Lee 
<sj...@apache.org<mailto:sj...@apache.org>>
Sent: Wednesday, November 30, 2016 3:24 PM
To: Junping Du
Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>; 
yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>; Vinod 
Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com<mailto:sj...@twitter.com>
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks for picking this up Junping!

I heard on email threads and offline discussions that there may be interests in 
porting some fixes from branch-2 to branch-2.8. I still feel that the bar 
should be pretty high to do that, but it might be good to have folks suggest 
some critical fixes that need to be released sooner. Thoughts?

On Tue, Nov 29, 2016 at 8:50 PM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com><mailto:j...@hortonworks.com<mailto:j...@hortonworks.com>>>
 wrote:
Hi folks,
 Per asked by Vinod offline, I would like to continue the effort to push 
2.8 release out in next several weeks. The plan is as following:
 - We only have 5 blockers and 3 critical issues 
(https://s.apache.org/6kwx) so far, and most of them are in good progress now. 
I put them all on apache ciwiki ( 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release) and will 
do daily monitor and update on these issues in next couple of weeks.
 - When all blocker/critical issues are gone (target EOD is Dec. 10), I 
will push out all major/minor issues that is open. So if you are working on 
some major issues and target for release in 2.8, please keep track of our 
release status and your work progress.
 - I will create RC immediately once our 2.8 target issues are clean 
(especially for blocker and critical issues) for voting process.

 I hope we can make it within year 2016, best before Xmas holiday. And I 
need help from all of you. :)


Thanks,

Junping


From: Vinod Kumar Vavilapalli 
<vino...@apache.org<mailto:vino...@apache.org><mailto:vino...@apache.org<mailto:vino...@apache.org>>>
Sent: Monday, July 25, 2016 2:57 PM
To: Jian He
Cc: 
mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org><mailto:mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>>;
 
yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org><mailto:yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>>;
 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org><mailto:hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>>;
 
common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org><mailto:common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>>
Subject: Re: [Release thread] 2.8.0 release activities

Thanks for all that great help moving this release forward, Jian, Sangjin, 
Wangda et al! Appreciate it.

The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last week.

I only see 9 blocker / critical fixes pending for 2.8.0 - 
https://issues.apache.org/jira/issues/?filter=12334985. Let’s do this!

Thanks
+Vinod

> On May 11, 2016, at 6:04 PM, Jian He 
> <j..

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Hi Sangjin and all,

 That sounds good. If anyone have priority fix to backport , please 
nominate it by following this email thread or ping me on JIRA directly. And I 
agree that we should keep in mind that our bar for 2.8 release should be high 
now as we want to move quickly on this release. Non-critical fixes can wait for 
next one.

  Any other comments and suggestions?


Thanks,


Junping



From: sjl...@gmail.com <sjl...@gmail.com> on behalf of Sangjin Lee 
<sj...@apache.org>
Sent: Wednesday, November 30, 2016 3:24 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Vinod Vavilapalli; 
Jian He; Wangda Tan; sj...@twitter.com
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks for picking this up Junping!

I heard on email threads and offline discussions that there may be interests in 
porting some fixes from branch-2 to branch-2.8. I still feel that the bar 
should be pretty high to do that, but it might be good to have folks suggest 
some critical fixes that need to be released sooner. Thoughts?

On Tue, Nov 29, 2016 at 8:50 PM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi folks,
 Per asked by Vinod offline, I would like to continue the effort to push 
2.8 release out in next several weeks. The plan is as following:
 - We only have 5 blockers and 3 critical issues 
(https://s.apache.org/6kwx) so far, and most of them are in good progress now. 
I put them all on apache ciwiki ( 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release) and will 
do daily monitor and update on these issues in next couple of weeks.
 - When all blocker/critical issues are gone (target EOD is Dec. 10), I 
will push out all major/minor issues that is open. So if you are working on 
some major issues and target for release in 2.8, please keep track of our 
release status and your work progress.
 - I will create RC immediately once our 2.8 target issues are clean 
(especially for blocker and critical issues) for voting process.

 I hope we can make it within year 2016, best before Xmas holiday. And I 
need help from all of you. :)


Thanks,

Junping


From: Vinod Kumar Vavilapalli <vino...@apache.org<mailto:vino...@apache.org>>
Sent: Monday, July 25, 2016 2:57 PM
To: Jian He
Cc: mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>; 
yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>
Subject: Re: [Release thread] 2.8.0 release activities

Thanks for all that great help moving this release forward, Jian, Sangjin, 
Wangda et al! Appreciate it.

The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last week.

I only see 9 blocker / critical fixes pending for 2.8.0 - 
https://issues.apache.org/jira/issues/?filter=12334985. Let’s do this!

Thanks
+Vinod

> On May 11, 2016, at 6:04 PM, Jian He 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
>
> For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs attention 
> for 2.8
>
> MAPREDUCE-6288
> YARN-1815
> YARN-4685
> YARN-4844
>
> The rest are either improvements or long-standing issues and does not qualify 
> release blocker, IMO.
> I think we’ll try to get these 4 jiras in asap. The rest will be on best 
> effort, resolve as much as possible and move them out if not resolved in time.
>
> Jian
>
> On May 11, 2016, at 5:37 PM, Wangda Tan 
> <wheele...@gmail.com<mailto:wheele...@gmail.com><mailto:wheele...@gmail.com<mailto:wheele...@gmail.com>>>
>  wrote:
>
> Sounds good to me :).
>
> Jian and I have looked at all existing 2.8.0 blockers and criticals today.
> To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
> out. Left comments on these JIRAs asked original owners, plan to update
> target version of these JIRAs early next week.
>
> Will keep this thread updated.
>
> Thanks,
> Wangda
>
>
> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org><mailto:sj...@apache.org<mailto:sj...@apache.org>>>
>  wrote:
>
> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
> report the list here. Then folks can chime in if you agree
>
> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
>
> Thanks,
> Sangjin
>
> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan 
> <wheele...@gmail.com<mailto:wheele...@gmail.com

[Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Hi folks,
 Per asked by Vinod offline, I would like to continue the effort to push 
2.8 release out in next several weeks. The plan is as following:
 - We only have 5 blockers and 3 critical issues 
(https://s.apache.org/6kwx) so far, and most of them are in good progress now. 
I put them all on apache ciwiki ( 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release) and will 
do daily monitor and update on these issues in next couple of weeks.
 - When all blocker/critical issues are gone (target EOD is Dec. 10), I 
will push out all major/minor issues that is open. So if you are working on 
some major issues and target for release in 2.8, please keep track of our 
release status and your work progress.
 - I will create RC immediately once our 2.8 target issues are clean 
(especially for blocker and critical issues) for voting process. 

 I hope we can make it within year 2016, best before Xmas holiday. And I 
need help from all of you. :)


Thanks,

Junping


From: Vinod Kumar Vavilapalli 
Sent: Monday, July 25, 2016 2:57 PM
To: Jian He
Cc: mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; common-...@hadoop.apache.org
Subject: Re: [Release thread] 2.8.0 release activities

Thanks for all that great help moving this release forward, Jian, Sangjin, 
Wangda et al! Appreciate it.

The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last week.

I only see 9 blocker / critical fixes pending for 2.8.0 - 
https://issues.apache.org/jira/issues/?filter=12334985. Let’s do this!

Thanks
+Vinod

> On May 11, 2016, at 6:04 PM, Jian He  wrote:
>
> For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs attention 
> for 2.8
>
> MAPREDUCE-6288
> YARN-1815
> YARN-4685
> YARN-4844
>
> The rest are either improvements or long-standing issues and does not qualify 
> release blocker, IMO.
> I think we’ll try to get these 4 jiras in asap. The rest will be on best 
> effort, resolve as much as possible and move them out if not resolved in time.
>
> Jian
>
> On May 11, 2016, at 5:37 PM, Wangda Tan 
> > wrote:
>
> Sounds good to me :).
>
> Jian and I have looked at all existing 2.8.0 blockers and criticals today.
> To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
> out. Left comments on these JIRAs asked original owners, plan to update
> target version of these JIRAs early next week.
>
> Will keep this thread updated.
>
> Thanks,
> Wangda
>
>
> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee 
> > wrote:
>
> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
> report the list here. Then folks can chime in if you agree
>
> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
>
> Thanks,
> Sangjin
>
> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan 
> > wrote:
>
> +1, we should close such staled JIRAs to avoid doing unnecessary checks
> for
> every releases.
>
> I'm working on reviewing YARN/MR critical/blocker patches currently, it
> gonna very helpful if someone else can help with reviewing Common/HDFS
> JIRAs.
>
> Thanks,
> Wangda
>
>
> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee 
> > wrote:
>
> Where do we stand in terms of closing out blocker/critical issues for
> 2.8.0? I still see 50 open JIRAs in Vinod's list:
> https://issues.apache.org/jira/issues/?filter=12334985
>
> But I see a lot of JIRAs with no patches or very stale patches. It
> would be
> a good exercise to come up with the list of JIRAs that we need to block
> 2.8.0 for and focus our attention on closing them out. Thoughts?
>
> Thanks,
> Sangjin
>
> On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 
> 
>
> wrote:
>
>
> On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> We are not converging - there’s still 58 more. I need help from the
> community in addressing / review 2.8.0 blockers. If folks can start
> with
> reviewing Patch available tickets, that’ll be great.
>
>
>
>
> I'm still doing the s3a stuff, other people testing and reviewing this
> stuff welcome.
>
> in particular, I could do with others playing with this patch of mine,
> which adds counters and things into S3a, based on the azure
> instrumentation
>
> https://issues.apache.org/jira/browse/HADOOP-13028
>
>
>
>
>
>
>
>


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To 

[jira] [Created] (YARN-5718) TimelineClient (and other places in YARN) shouldn't over-write HDFS client retry settings which could cause unexpected behavior

[Junping Du (JIRA)]

Junping Du created YARN-5718:


 Summary: TimelineClient (and other places in YARN) shouldn't 
over-write HDFS client retry settings which could cause unexpected behavior
 Key: YARN-5718
 URL: https://issues.apache.org/jira/browse/YARN-5718
 Project: Hadoop YARN
  Issue Type: Bug
  Components: timelineclient, resourcemanager
Reporter: Junping Du
Assignee: Junping Du


In one HA cluster, after NN failed over, we noticed that job is getting failed 
as TimelineClient failed to retry connection to proper NN. This is because we 
are overwrite hdfs client settings that hard code retry policy to be enabled 
that conflict NN failed-over case - hdfs client should fail fast so can retry 
on another NN.
We shouldn't assume any retry policy for hdfs client at all places in YARN. 
This should keep consistent with HDFS settings that has different retry polices 
in different deployment case. Thus, we should clean up these hard code settings 
in YARN, include: FileSystemTimelineWriter, FileSystemRMStateStore and 
FileSystemNodeLabelsStore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Junping Du]

+1 binding, based on:

- Download and build from source, check signatures

- Deploy a single node cluster and run some simple jobs, like sleep, PI, etc.

Thanks Sangjin and Chris for working on 2.6.5 release!

Thanks,

Junping

From: larry mccay 
Sent: Saturday, October 08, 2016 3:43 AM
To: Karthik Kambatla
Cc: Yongjun Zhang; Sangjin Lee; Andrew Wang; Wangda Tan; Zhihai Xu; Masatake 
Iwasaki; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

+1 (non-binding)


* Downloaded and verified signatures

* Built from source

* Deployed a standalone cluster

* Tested HDFS commands and job submit

* Tested webhdfs through Apache Knox

On Fri, Oct 7, 2016 at 10:35 PM, Karthik Kambatla 
wrote:

> Thanks for putting the RC together, Sangjin.
>
> +1 (binding)
>
> Built from source, deployed pseudo distributed cluster and ran some example
> MR jobs.
>
> On Fri, Oct 7, 2016 at 6:01 PM, Yongjun Zhang  wrote:
>
> > Hi Sangjin,
> >
> > Thanks a lot for your work here.
> >
> > My +1 (binding).
> >
> > - Downloaded both binary and src tarballs
> > - Verified md5 checksum and signature for both
> > - Build from source tarball
> > - Deployed 2 pseudo clusters, one with the released tarball and the other
> > with what I built from source, and did the following on both:
> > - Run basic HDFS operations, and distcp jobs
> > - Run pi job
> > - Examined HDFS webui, YARN webui.
> >
> > Best,
> >
> > --Yongjun
> >
> > > > > * verified basic HDFS operations and Pi job.
> > > > > * Did a sanity check for RM and NM UI.
> >
> >
> > On Fri, Oct 7, 2016 at 5:08 PM, Sangjin Lee  wrote:
> >
> > > I'm casting my vote: +1 (binding)
> > >
> > > Regards,
> > > Sangjin
> > >
> > > On Fri, Oct 7, 2016 at 3:12 PM, Andrew Wang 
> > > wrote:
> > >
> > > > Thanks to Chris and Sangjin for working on this release.
> > > >
> > > > +1 binding
> > > >
> > > > * Verified signatures
> > > > * Built from source tarball
> > > > * Started HDFS and did some basic ops
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > On Fri, Oct 7, 2016 at 2:50 PM, Wangda Tan 
> > wrote:
> > > >
> > > > > Thanks Sangjin for cutting this release!
> > > > >
> > > > > +1 (Binding)
> > > > >
> > > > > - Downloaded binary tar ball and setup a single node cluster.
> > > > > - Submit a few applications and which can successfully run.
> > > > >
> > > > > Thanks,
> > > > > Wangda
> > > > >
> > > > >
> > > > > On Fri, Oct 7, 2016 at 10:33 AM, Zhihai Xu  >
> > > > wrote:
> > > > >
> > > > > > Thanks Sangjin for creating release 2.6.5 RC1.
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > * Downloaded and built from source
> > > > > > * Verified md5 checksums and signature
> > > > > > * Deployed a pseudo cluster
> > > > > > * verified basic HDFS operations and Pi job.
> > > > > > * Did a sanity check for RM and NM UI.
> > > > > >
> > > > > > Thanks
> > > > > > zhihai
> > > > > >
> > > > > > On Fri, Oct 7, 2016 at 8:16 AM, Sangjin Lee 
> > > wrote:
> > > > > >
> > > > > > > Thanks Masatake!
> > > > > > >
> > > > > > > Today's the last day for this vote, and I'd like to ask you to
> > try
> > > > out
> > > > > > the
> > > > > > > RC and vote on this today. So far there has been no binding
> vote.
> > > > > Thanks
> > > > > > > again.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Sangjin
> > > > > > >
> > > > > > > On Fri, Oct 7, 2016 at 6:45 AM, Masatake Iwasaki <
> > > > > > > iwasak...@oss.nttdata.co.jp> wrote:
> > > > > > >
> > > > > > > > +1(non-binding)
> > > > > > > >
> > > > > > > > * verified signature and md5.
> > > > > > > > * built with -Pnative on CentOS6 and OpenJDK7.
> > > > > > > > * built documentation and skimmed the contents.
> > > > > > > > * built rpms by bigtop and ran smoke-tests of hdfs, yarn and
> > > > > mapreduce
> > > > > > on
> > > > > > > > 3-node cluster.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Masatake Iwasaki
> > > > > > > >
> > > > > > > > On 10/3/16 09:12, Sangjin Lee wrote:
> > > > > > > >
> > > > > > > >> Hi folks,
> > > > > > > >>
> > > > > > > >> I have pushed a new release candidate (R1) for the Apache
> > Hadoop
> > > > > 2.6.5
> > > > > > > >> release (the next maintenance release in the 2.6.x release
> > > line).
> > > > > RC1
> > > > > > > >> contains fixes to CHANGES.txt, and is otherwise identical to
> > > RC0.
> > > > > > > >>
> > > > > > > >> Below are the details of this release candidate:
> > > > > > > >>
> > > > > > > >> The RC is available for validation at:
> > > > > > > >> http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/.
> > > > > > > >>
> > > > > > > >> The RC tag in git is release-2.6.5-RC1 and its git commit is
> > > > > > > >> 

Re: [REMINDER] How to set fix versions when committing

[Junping Du]

Hi Andrew and all,
 Thanks for the notice on the change. I still concern this rule change may 
cause some confusion from conflicting against our previous rule - no need to 
set trunk version if it is landing on 2.x branch. As we can see, there are 4 
cases of version setting for JIRA landing on trunk and branch-2 now:
1. JIRA with fixed version set to 2.x only before 3.0.0-alpha1 cut off. 
2. JIRA with fixed version set to 2.x only after 3.0.0-alpha1 cut off.
3. JIRA with fixed version set to 2.x and 3.0.0-alpha1 after 3.0.0-alpha1 cut 
off.
4. JIRA with fixed version set to 2.x and 3.0.0-alpha2 after 3.0.0-alpha1 cut 
off

Case 3 and 4 can be easily distinguished, but case 1 and 2 is against our rule 
change here and hard to differentiate unless we want to mark all previous JIRA 
with fixed version for 2.x only to include 3.0.0-alpha1/3.0.0-alpha2. That's a 
tremendous effort and I doubt this should be our option.
Or, my preference is: we should update our rule here a bit to assume all JIRA 
marked fixed version to 2.x only get landed to 3.0.0-alpha1 but in the 
meanwhile, we monitor all JIRAs come after 3.0.0-alpha1 cut off to include 
3.0.0-alpha2 (or latest trunk version).
Thoughts?


Thanks,

Junping


From: Andrew Wang 
Sent: Tuesday, August 30, 2016 2:57 AM
To: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: [REMINDER] How to set fix versions when committing

Hi all,

I finished the bulk fix version update and just rebranched
branch-3.0.0-alpha1 off of trunk. So, a reminder that the procedure for
setting fix versions has changed slightly from before.

Everything is fully detailed here, the example in particular should help
clarify things:

https://hadoop.apache.org/versioning.html

The short of it though is that if a JIRA is going into trunk or
branch-3.0.0-alpha1, it should also have a 3.0.0-alpha1 or 3.0.0-alpha2
fixVersion set.

Thanks,
Andrew

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Junping Du]

Thanks Vinod for creating new RC for 2.7.3 release.

+1 (binding) based on following verifications:

- Download src and binary tar ball and verify signature (gpg --verify).

- Build from source Java 1.8.0_31-b13 on Mac native successfully.

- Build from source with Java 1.7.0_79-b15 on Ubuntu VM successfully.

- Deploy a pseudo cluster with running some simple MR jobs, like: sleep, pi, 
teragen/terasort, etc. All finished successfully.

- Check RM/NM web UI and go through some pages like: Scheduler, Nodes, 
Application, etc. Everything works correct.

- Verify basic yarn CLIs, check version (for hadoop and yarn), classpath, 
application list, etc. all seems to work fine.

Thanks,

Junping


From: Vinod Kumar Vavilapalli 
Sent: Thursday, August 18, 2016 3:05 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Cc: Vinod Kumar Vavilapalli
Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2

Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - few more issues with RC1 forced a RC2 [2]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout

[Junping Du (JIRA)]

Junping Du created YARN-5536:


 Summary: Multiple format support (JSON, etc.) for exclude node 
file in NM graceful decommission with timeout
 Key: YARN-5536
 URL: https://issues.apache.org/jira/browse/YARN-5536
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Junping Du
Priority: Blocker


Per discussion in YARN-4676, we agree that multiple format (other than xml) 
should be supported to decommission nodes with timeout values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Junping Du]

I think Allen's previous comments are very misleading. 
In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) 
shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, 
daemon's restart, etc.) should get flexible for landing. Otherwise, how could 
52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could 
get landed on branch-2 after 2.2.0 release? Most of them are already released. 

Thanks,

Junping

From: Vinod Kumar Vavilapalli 
Sent: Wednesday, August 17, 2016 9:29 PM
To: Allen Wittenauer
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

I always look at CHANGES.txt entries for incompatible-changes and this JIRA 
obviously wasn’t there.

Anyways, this shouldn’t be in any of branch-2.* as committers there clearly 
mentioned that this is an incompatible change.

I am reverting the patch from branch-2* .

Thanks
+Vinod

> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer  
> wrote:
>
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli  
>> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
>> 
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org 
>>  at 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
>> 
>>
>> The release-notes are inside the tar-balls at location 
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
>>  for 
>> your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. 
>> This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
>> 
>> [2]: 2.7.3 release plan: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>> 
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Junping Du]

>From my quick understanding, HDFS-9395 is more like a bug fix and improvement 
>for audit logging instead of incompatible changes. We mark incompatible 
>probably because the audit log behavior could be corrected/updated in some 
>exception cases. I think it still belongs to 2.7.3 scope. 
Kuhu and Kihwal, any comments here?


Thanks,

Junping 

From: Allen Wittenauer 
Sent: Wednesday, August 17, 2016 5:29 AM
To: common-...@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

-1

HDFS-9395 is an incompatible change:

a) Why is not marked as such in the changes file?
b) Why is an incompatible change in a micro release, much less a minor?
c) Where is the release note for this change?


> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli  
> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: 
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
> 
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org 
>  at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
> 
>
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
>  for 
> your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
> 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
> release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
> 
> [2]: 2.7.3 release plan: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Release thread] 2.6.5 release activities

[Junping Du]

I am OK with going forward for 2.6.5 release given some people may not be ready 
for migration to 2.7 and we have done some preparation work already. Thanks 
Chris for pushing the release work forward!
About future releases of 2.6 (after 2.6.5) or any other legacy releases, I 
think it worth more discussions. I disagree with what Allen said below - Java 6 
support should be a solid reason for branch-2.6 release keep going. In this 
community, we are so aggressive to drop Java 7 support in 3.0.x release. Here, 
why we are so conservative to keep releasing new bits to support Java 6? Our 
release effort, although driven by different person, should be consistent 
logically. 
I don't think we have clear rules/polices about EOL of legacy releases before. 
This is a bit off topic here. Will start a new thread later for more discussion.

Thanks,

Junping


From: Chris Trezzo <ctre...@gmail.com>
Sent: Friday, August 12, 2016 12:42 AM
To: Karthik Kambatla
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Release thread] 2.6.5 release activities

Thanks everyone for the discussion! Based on what has been said in this
thread, I will move forward on the preparation efforts (working with
Sangjin and the community) for a 2.6.5 release. If there is a strong
objection to this, please let us know.

I see three initial tasks going forward:

1. Fix the pre-commit build for branch-2.6. I am not entirely sure what is
involved here, but will start looking into it using HADOOP-12800 as a
starting point.

2. Look through the unresolved issues still targeted to 2.6.5 and
resolve/re-target as necessary. There are currently 17 of them:
https://issues.apache.org/jira/issues/?jql=(project%20%
3D%20%22Hadoop%20Common%22%20OR%20project%20%3D%20%22Hadoop%20YARN%22%20OR%
20project%20%3D%20%22Hadoop%20Map%2FReduce%22%20OR%
20project%20%3D%20%22Hadoop%20HDFS%22)%20AND%20(
fixVersion%20%3D%202.6.5%20OR%20%22Target%20Version%2Fs%22%
20%3D%202.6.5)%20AND%20(status%20!%3D%20Resolved%20AND%20status%20!%3D%
20Closed)%20ORDER%20BY%20status%20ASC

3. Look through the backport candidates in the spreadsheet in more detail
and backport/drop as necessary. There are currently 15 of them:
https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC
8hTRUemHvYPPzY/edit?usp=sharing

Finally, I think it would be awesome to continue the release end-of-life
discussion. As we get better and better at release cadence it is going to
be really helpful to have an established EOL policy. It will be less
confusing for contributors and help set clear expectations for end users as
to when they need to start making reasonable migration plans, especially if
they want to stay on a well supported release line. I would be happy to
help with this effort as well. It would be great if we could leverage that
discussion and have EOL information for the 2.6 line when we release 2.6.5.

As always, please let me know if I have missed something.

Thanks!
Chris Trezzo

On Thu, Aug 11, 2016 at 1:03 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> Since there is sufficient interest in 2.6.5, we should probably do it. All
> the reasons Allen outlines make sense.
>
> That said, Junping brings up a very important point that we should think of
> for future releases. For a new user or a user that does not directly
> contribute to the project, more stable releases make it hard to pick from.
>
> As Chris T mentioned, the notion of EOL for our releases seems like a good
> idea. However, to come up with any EOLs, we need to have some sort of
> cadence for the releases. While this is hard for major releases (big bang,
> potentially incompatible features), it should be doable for minor releases.
>
> How do people feel about doing a minor release every 6 months, with
> follow-up maintenance releases every 2 months until the next minor and as
> needed after that? That way, we could EOL a minor release a year after its
> initial release? In the future, we could consider shrinking this window. In
> addition to the EOL, this also makes our releases a little more predictable
> for both users and vendors. Note that 2.6.0 went out almost 2 years ago and
> we haven't had a new minor release in 14 months. I am happy to start
> another DISCUSS thread around this if people think it is useful.
>
> Thanks
> Karthik
>
> On Thu, Aug 11, 2016 at 12:50 PM, Allen Wittenauer <
> a...@effectivemachines.com
> > wrote:
>
> >
> > > On Aug 11, 2016, at 8:10 AM, Junping Du <j...@hortonworks.com> wrote:
> > >
> > > Allen, to be clear, I am not against any branch release effort here.
> > However,
> >
> > "I'm not an X but "
> >
> > > as RM for previous releases 2.6.3 and 2.6.4, I f

Re: [Release thread] 2.6.5 release activities

[Junping Du]

Allen, to be clear, I am not against any branch release effort here. However, 
as RM for previous releases 2.6.3 and 2.6.4, I feel to have responsibility to 
take care branch-2.6 together with other RMs (Vinod and Sangjin) on this branch 
and understand current gap - especially, to get consensus from community on the 
future plan for 2.6.x.
Our bylaw give us freedom for anyone to do release effort, but our bylaw 
doesn't stop our rights for reasonable question/concern on any release plan. As 
you mentioned below, people can potentially fire up branch-1 release effort. 
But if you call a release plan tomorrow for branch-1, I cannot imagine nobody 
will question on that effort. Isn't it? 
Let's keep discussions on releasing 2.6.5 more technical. IMO, to make 2.6.5 
release more reasonable, shouldn't we check following questions first?
1. Do we have any significant issues that should land on 2.6.5 comparing with 
2.6.4?
2. If so, any technical reasons (like: upgrade is not smoothly, performance 
downgrade, incompatibility with downstream projects, etc.) to stop our users to 
move from 2.6.4 to 2.7.2/2.7.3?
I believe having good answer on these questions can make our release plan more 
reasonable to the whole community. More thoughts?

Thanks,

Junping

From: Allen Wittenauer <a...@effectivemachines.com>
Sent: Thursday, August 11, 2016 3:13 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Release thread] 2.6.5 release activities

> On Aug 11, 2016, at 5:59 AM, Junping Du <j...@hortonworks.com> wrote:
>
>  These comments are more like wishes but not giving more clarifications 
> on the needs. I would like to hear more specific reasons to not move to 2.7.x 
> releases but prefer to upgrade to 2.6.5. If the only reason is about 
> expectation management, I think we should claim 2.6.5 is the last branch-2.6 
> release after this release work, otherwise people would expect us to maintain 
> this branch forever which is impossible and unnecessary. Thoughts?

The bylaws[*] are such that if community members want to spend their 
time working on a branch, there isn't much to prevent that other than the PMC 
voting down the release of that branch or removing the committers working on 
that branch.  As has been pointed out to me many times, one can't dictate where 
others spend their volunteer time.  If they want to spend their efforts on 
branch-2.6, they can.  If that comes at the detriment of releases around 
branch-2.7 or branch-2.8 or even trunk, then so be it. Technically, someone 
could still fire up a branch-1 release.  Given the numbers of committers and 
PMC members as listed on the main ASF website (not the list on project one), we 
should have more than enough people to do all this work anyway.

* - of course, there's a few bylaws that aren't really enforced, so maybe even 
this isn't true?

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Release thread] 2.6.5 release activities

[Junping Du]

Hi Chris,

  Thanks for your response!

  I think I could miss the thread discussion of "[DISCUSS] 2.6.x line 
releases" for something reason. I checked the discussion - Sean claimed that 
HBase community needs 2.6.5, Zhe said they are using 2.6.x releases and Akira 
said that are over new 50 commits land on branch-2.6 since 2.6.4. Do I miss any 
comments there?

  These comments are more like wishes but not giving more clarifications on 
the needs. I would like to hear more specific reasons to not move to 2.7.x 
releases but prefer to upgrade to 2.6.5. If the only reason is about 
expectation management, I think we should claim 2.6.5 is the last branch-2.6 
release after this release work, otherwise people would expect us to maintain 
this branch forever which is impossible and unnecessary. Thoughts?


Thanks,


Junping



From: Chris Trezzo <ctre...@gmail.com>
Sent: Wednesday, August 10, 2016 9:30 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: [Release thread] 2.6.5 release activities

Thanks Jason and Junping for the comments! I will update the spreadsheet for 
HADOOP-13362 and YARN-4794.

As for continuing 2.6.x releases, please see the discussion in the "[DISCUSS] 
2.6.x line releases" thread. Sean, Akira and Zhe all expressed interest in 
additional 2.6.x releases. I started this thread based off of that interest. I 
understand there is a burden to maintaining a large number of branches. I am 
not sure what the community's end-of-life policy is, but maybe we can issue a 
warning with the 2.6.5 release stating when we will stop maintaining the 
release line. This at least gives users some time to make migration plans to a 
newer version.

On Wed, Aug 10, 2016 at 9:36 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Thanks Chris for bring up this discussion.
Before we going to detail discussion of releasing 2.6.5, I have a quick 
question here: do we think it is necessary to continue to release branch-2.6, 
like 2.6.5, etc after 2.7 is out for more than 1 year. Any reason to not 
suggest users to upgrade to 2.7.3 releases for latest fixes which is in 
releasing now?
My major concern on more release efforts on legacy branches is the same with my 
comments on other release plan before - it seems too many releases trains get 
planned at the same time window (2.6.x, 2.7.x, 2.8, 3.0-alpha, 3.1-beta, etc.). 
Not only user could get confusing on this, but also I suspect we don't have so 
many bandwidth in community to push forward so these releases in high quality 
during the same time window - just like Chris Douglas mentioned in another 
email thread on committer activity and bandwidth. IMO, may be it is better to 
focus on limited number of releases and move them faster?

BTW, I agree with Jason that HADOOP-13362 is not needed for branch-2.6 unless 
we backport container metrics related patches there.


Thanks,

Junping

From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
Sent: Wednesday, August 10, 2016 4:14 PM
To: Chris Trezzo; 
common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>; 
yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>
Subject: Re: [Release thread] 2.6.5 release activities

Thanks for organizing this, Chris!
I don't believe HADOOP-13362 is needed since it's related to ContainerMetrics.  
ContainerMetrics weren't added until 2.7 by YARN-2984.
YARN-4794 looks applicable to 2.6.  The change drops right in except it has 
JDK7-isms (multi-catch clause), so it needs a slight change.

Jason

  From: Chris Trezzo <ctre...@gmail.com<mailto:ctre...@gmail.com>>
 To: "common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>" 
<common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
"mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>" 
<mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>>; 
"yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>" 
<yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>>
 Sent: Tuesday, August 9, 2016 7:32 PM
 Subject: [Release thread] 2.6.5 release activities

Based on the sentiment in the "[DISCUSS] 2.6.x line releases" thread, I
have moved forward with some of the initial effort in creating a 2.6.5
release. I am forking this thread so we have a dedicated 2.6.5 release
thread.

I have gone through the git logs and gathered a list of JIRAs that are in
branch-2.7 but 

Re: [Release thread] 2.6.5 release activities

[Junping Du]

Thanks Chris for bring up this discussion. 
Before we going to detail discussion of releasing 2.6.5, I have a quick 
question here: do we think it is necessary to continue to release branch-2.6, 
like 2.6.5, etc after 2.7 is out for more than 1 year. Any reason to not 
suggest users to upgrade to 2.7.3 releases for latest fixes which is in 
releasing now?
My major concern on more release efforts on legacy branches is the same with my 
comments on other release plan before - it seems too many releases trains get 
planned at the same time window (2.6.x, 2.7.x, 2.8, 3.0-alpha, 3.1-beta, etc.). 
Not only user could get confusing on this, but also I suspect we don't have so 
many bandwidth in community to push forward so these releases in high quality 
during the same time window - just like Chris Douglas mentioned in another 
email thread on committer activity and bandwidth. IMO, may be it is better to 
focus on limited number of releases and move them faster?

BTW, I agree with Jason that HADOOP-13362 is not needed for branch-2.6 unless 
we backport container metrics related patches there.


Thanks,

Junping

From: Jason Lowe 
Sent: Wednesday, August 10, 2016 4:14 PM
To: Chris Trezzo; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Release thread] 2.6.5 release activities

Thanks for organizing this, Chris!
I don't believe HADOOP-13362 is needed since it's related to ContainerMetrics.  
ContainerMetrics weren't added until 2.7 by YARN-2984.
YARN-4794 looks applicable to 2.6.  The change drops right in except it has 
JDK7-isms (multi-catch clause), so it needs a slight change.

Jason

  From: Chris Trezzo 
 To: "common-...@hadoop.apache.org" ; 
hdfs-...@hadoop.apache.org; "mapreduce-...@hadoop.apache.org" 
; "yarn-dev@hadoop.apache.org" 

 Sent: Tuesday, August 9, 2016 7:32 PM
 Subject: [Release thread] 2.6.5 release activities

Based on the sentiment in the "[DISCUSS] 2.6.x line releases" thread, I
have moved forward with some of the initial effort in creating a 2.6.5
release. I am forking this thread so we have a dedicated 2.6.5 release
thread.

I have gone through the git logs and gathered a list of JIRAs that are in
branch-2.7 but are missing from branch-2.6. I limited the diff to issues
with a commit date after 1/26/2016. I did this because 2.6.4 was cut from
branch-2.6 around that date (http://markmail.org/message/xmy7ebs6l3643o5e)
and presumably issues that were committed to branch-2.7 before then were
already looked at as part of 2.6.4.

I have collected these issues in a spreadsheet and have given them an
initial triage on whether they are candidates for a backport to 2.6.5. The
spreadsheet is sorted by the status of the issues with the potential
backport candidates at the top. Here is a link to the spreadsheet:
https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC8hTRUemHvYPPzY/edit?usp=sharing

As of now, I have identified 16 potential backport candidates. Please take
a look at the list and let me know if there are any that you think should
not be on the list, or ones that you think I have missed. This was just an
initial high-level triage, so there could definitely be issues that are
miss-labeled.

As a side note: we still need to look at the pre-commit build for 2.6 and
follow up with an addendum for HADOOP-12800.

Thanks everyone!
Chris Trezzo


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Junping Du]

I think that incompatible API between 3.0.0-alpha and 3.1.0-beta is something 
less confusing than incompatible between 2.8/2.9 and 2.98.x alphas/2.99.x betas.
Why not just follow our previous practice in the beginning of branch-2? we can 
have 3.0.0-alpha, 3.1.0-alpha/beta, but once when we are finalizing our APIs, 
we should bump up trunk version to 4.x for landing new incompatible changes.

Thanks,

Junping

From: Karthik Kambatla 
Sent: Monday, August 08, 2016 6:54 PM
Cc: common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [DISCUSS] Release numbering semantics with concurrent (>2) 
releases [Was Setting JIRA fix versions for 3.0.0 releases]

I like the 3.0.0-alphaX approach primarily for simpler understanding of
compatibility guarantees. Calling 3.0.0 alpha and 3.1.0 beta is confusing
because, it is not immediately clear that 3.0.0 and 3.1.0 could be
incompatible in APIs.

I am open to something like 2.98.x for alphas and 2.99.x for betas leading
to a 3.0.0 GA. I have seen other projects use this without causing much
confusion.

On Thu, Aug 4, 2016 at 6:01 PM, Konstantin Shvachko 
wrote:

> On Thu, Aug 4, 2016 at 11:20 AM, Andrew Wang 
> wrote:
>
> > Hi Konst, thanks for commenting,
> >
> > On Wed, Aug 3, 2016 at 11:29 PM, Konstantin Shvachko <
> shv.had...@gmail.com
> > > wrote:
> >
> >> 1. I probably missed something but I didn't get it how "alpha"s made
> >> their way into release numbers again. This was discussed on several
> >> occasions and I thought the common perception was to use just three
> level
> >> numbers for release versioning and avoid branding them.
> >> It is particularly confusing to have 3.0.0-alpha1 and 3.0.0-alpha2. What
> >> is alphaX - fourth level? I think releasing 3.0.0 and setting trunk to
> >> 3.1.0 would be perfectly in line with our current release practices.
> >>
> >
> > We discussed release numbering a while ago when discussing the release
> > plan for 3.0.0, and agreed on this scheme. "-alphaX" is essentially a
> > fourth level as you say, but the intent is to only use it (and "-betaX")
> in
> > the leadup to 3.0.0.
> >
> > The goal here is clarity for end users, since most other enterprise
> > software uses a a.0.0 version to denote the GA of a new major version.
> Same
> > for a.b.0 for a new minor version, though we haven't talked about that
> yet.
> > The alphaX and betaX scheme also shares similarity to release versioning
> of
> > other enterprise software.
> >
>
> As you remember we did this (alpha, beta) for Hadoop-2 and I don't think it
> went well with user perception.
> Say release 2.0.5-alpha turned out to be quite good even though still
> branded "alpha", while 2.2 was not and not branded.
> We should move a release to stable, when people ran it and agree it is GA
> worthy. Otherwise you never know.
>
>
> >
> >> 2. I do not see any confusions with releasing 2.8.0 after 3.0.0.
> >> The release number is not intended to reflect historical release
> >> sequence, but rather the point in the source tree, which it was branched
> >> off. So one can release 2.8, 2.9, etc. after or before 3.0.
> >>
> >
> > As described earlier in this thread, the issue here is setting the fix
> > versions such that the changelog is a useful diff from a previous
> version,
> > and also clear about what changes are present in each branch. If we do
> not
> > order a specific 2.x before 3.0, then we don't know what 2.x to diff
> from.
> >
>
> So the problem is in determining the latest commit, which was not present
> in the last release, when the last release bears higher number than the one
> being released.
> Interesting problem. Don't have a strong opinion on that. I guess it's OK
> to have overlapping in changelogs.
> As long as we keep following the rule that commits should be made to trunk
> first and them propagated to lower branches until the target branch is
> reached.
>
>
> >
> >> 3. I agree that current 3.0.0 branch can be dropped and re-cut. We may
> >> think of another rule that if a release branch is not released in 3
> month
> >> it should be abandoned. Which is applicable to branch 2.8.0 and it is
> too
> >> much work syncing it with branch-2.
> >>
> >> Time-based rules are tough here. I'd prefer we continue to leave this up
> > to release managers. If you think we should recut branch-2.8, recommend
> > pinging Vinod and discussing on a new thread.
> >
>
> Not recut, but abandon 2.8.0. And Vinod (or anybody who volunteers to RM)
> can recut  from the desired point.
> People were committing to branch-2 and branch-2.8 for months. And they are
> out of sync anyways. So what's the point of the extra commit.
> Probably still a different thread.
>
> Thanks,
> --Konst
>

-
To unsubscribe, e-mail: 

[jira] [Created] (YARN-5475) Test failed for TestAggregatedLogFormat on trunk

[Junping Du (JIRA)]

Junping Du created YARN-5475:


 Summary: Test failed for TestAggregatedLogFormat on trunk
 Key: YARN-5475
 URL: https://issues.apache.org/jira/browse/YARN-5475
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Junping Du


Tests run: 3, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 1.114 sec <<< 
FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat
testReadAcontainerLogs1(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat)
  Time elapsed: 0.012 sec  <<< ERROR!
java.io.IOException: Unable to create directory : 
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/TestAggregatedLogFormat/testReadAcontainerLogs1/srcFiles/application_1_0001/container_1_0001_01_01/subDir
at 
org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.getOutputStreamWriter(TestAggregatedLogFormat.java:403)
at 
org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.writeSrcFile(TestAggregatedLogFormat.java:382)
at 
org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testReadAcontainerLog(TestAggregatedLogFormat.java:211)
at 
org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testReadAcontainerLogs1(TestAggregatedLogFormat.java:185)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5416) TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped

[Junping Du (JIRA)]

Junping Du created YARN-5416:


 Summary: TestRMRestart#testRMRestartWaitForPreviousAMToFinish 
failed intermittently due to not wait SchedulerApplicationAttempt to be stopped
 Key: YARN-5416
 URL: https://issues.apache.org/jira/browse/YARN-5416
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Junping Du
Assignee: Junping Du
Priority: Minor


The test failure stack is:
Running org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
Tests run: 54, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 385.338 sec 
<<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
testRMRestartWaitForPreviousAMToFinish[0](org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
  Time elapsed: 43.134 sec  <<< FAILURE!
java.lang.AssertionError: AppAttempt state is not correct (timedout) 
expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at 
org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:86)
at 
org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:594)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.launchAM(TestRMRestart.java:1008)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartWaitForPreviousAMToFinish(TestRMRestart.java:530)

This is due to the same issue that partially fixed in YARN-4968



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5311) Document graceful decommission CLI and usage

[Junping Du (JIRA)]

Junping Du created YARN-5311:


 Summary: Document graceful decommission CLI and usage
 Key: YARN-5311
 URL: https://issues.apache.org/jira/browse/YARN-5311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: documentation
Reporter: Junping Du
Assignee: Junping Du






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Increased use of feature branches

[Junping Du]

Comparing with advantages, I believe the disadvantages of shipping any releases 
directly from trunk are more obvious and significant:
- A lot of commits (incompatible, risky, uncompleted feature, etc.) have to 
wait to commit to trunk or put into a separated branch that could delay feature 
development progress as additional vote process get involved even the feature 
is simple and harmless.

- These commits left in separated branches are isolated and get more chance to 
conflict each other, and more bugs could be involved due to conflicts and/or 
less eyes watching/bless on isolated branches.

- More unnecessary arguments/debates will happen on if some commits should land 
on trunk or a separated branch, just like what we have recently.

- Because branches will get increased massively, more community efforts will be 
spent on review & vote for branches merge that means less effort will be spent 
on other commits review given our review bandwidth is quite short so far.

- For small feature with only 1 or 2 commits, that need three +1 from PMCs will 
increase the bar largely for contributors who just start to contribute on 
Hadoop features but no such sufficient support.

Given these concerns, I am open to other options, like: proposed by Vinod or 
Chris, but rather than to release anything directly from trunk.

- This point doesn't necessarily need to be resolved now though, since again 
we're still doing alphas.
No. I think we have to settle down this first. Without a common agreed and 
transparent release process and branches in community, any release (alpha, 
beta) bits is only called a private release but not a official apache hadoop 
release (even alpha).


Thanks,

Junping

From: Karthik Kambatla 
Sent: Friday, June 10, 2016 7:49 AM
To: Andrew Wang
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Increased use of feature branches

Thanks for restarting this thread Andrew. I really hope we can get this
across to a VOTE so it is clear.

I see a few advantages shipping from trunk:

   - The lack of need for one additional backport each time.
   - Feature rot in trunk

Instead of creating branch-3, I recommend creating branch-3.x so we can
continue doing 3.x releases off branch-3 even after we move trunk to 4.x (I
said it :))

On Thu, Jun 9, 2016 at 11:12 PM, Andrew Wang 
wrote:

> Hi all,
>
> On a separate thread, a question was raised about 3.x branching and use of
> feature branches going forward.
>
> We discussed this previously on the "Looking to a Hadoop 3 release" thread
> that has spanned the years, with Vinod making this proposal (building on
> ideas from others who also commented in the email thread):
>
>
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201604.mbox/browser
>
> Pasting here for ease:
>
> On an unrelated note, offline I was pitching to a bunch of
> contributors another idea to deal
> with rotting trunk post 3.x: *Make 3.x releases off of trunk directly*.
>
> What this gains us is that
>  - Trunk is always nearly stable or nearly ready for releases
>  - We no longer have some code lying around in some branch (today’s
> trunk) that is not releasable
> because it gets mixed with other undesirable and incompatible changes.
>  - This needs to be coupled with more discipline on individual
> features - medium to to large
> features are always worked upon in branches and get merged into trunk
> (and a nearing release!)
> when they are ready
>  - All incompatible changes go into some sort of a trunk-incompat
> branch and stay there till
> we accumulate enough of those to warrant another major release.
>
> Regarding "trunk-incompat", since we're still in the alpha stage for 3.0.0,
> there's no need for this branch yet. This aspect of Vinod's proposal was
> still under a bit of discussion; Chris Douglas though we should cut a
> branch-3 for the first 3.0.0 beta, which aligns with my original thinking.
> This point doesn't necessarily need to be resolved now though, since again
> we're still doing alphas.
>
> What we should get consensus on is the goal of keeping trunk stable, and
> achieving that by doing more development on feature branches and being
> judicious about merges. My sense from the Hadoop 3 email thread (and the
> more recent one on the async API) is that people are generally in favor of
> this.
>
> We're just about ready to do the first 3.0.0 alpha, so would greatly
> appreciate everyone's timely response in this matter.
>
> Thanks,
> Andrew
>

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-5217) Close FileInputStream in NMWebServices#getLogs in branch-2.8

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du resolved YARN-5217.
--
Resolution: Duplicate

> Close FileInputStream in NMWebServices#getLogs in branch-2.8
> 
>
> Key: YARN-5217
> URL: https://issues.apache.org/jira/browse/YARN-5217
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 2.8.0
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: YARN-5217.branch-2.8.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-5199, we close LogReader in in 
> AHSWebServices#getStreamingOutput and FileInputStream in 
> NMWebServices#getLogs. We should do the same thing in branch-2.8.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5214) Pending on synchronized method DirectoryCollection#checkDirs can hang NM's NodeStatusUpdater

[Junping Du (JIRA)]

Junping Du created YARN-5214:


 Summary: Pending on synchronized method 
DirectoryCollection#checkDirs can hang NM's NodeStatusUpdater
 Key: YARN-5214
 URL: https://issues.apache.org/jira/browse/YARN-5214
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Reporter: Junping Du
Assignee: Junping Du
Priority: Critical


In one cluster, we notice NM's heartbeat to RM is suddenly stopped and wait a 
while and marked LOST by RM. From the log, the NM daemon is still running, but 
jstack hints NM's NodeStatusUpdater thread get blocked:
1.  Node Status Updater thread get blocked by 0x8065eae8 
{noformat}
"Node Status Updater" #191 prio=5 os_prio=0 tid=0x7f0354194000 nid=0x26fa 
waiting for monitor entry [0x7f035945a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.getFailedDirs(DirectoryCollection.java:170)
- waiting to lock <0x8065eae8> (a 
org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection)
at 
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getDisksHealthReport(LocalDirsHandlerService.java:287)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService.getHealthReport(NodeHealthCheckerService.java:58)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:389)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:83)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:643)
at java.lang.Thread.run(Thread.java:745)
{noformat}

2. The actual holder of this lock is DiskHealthMonitor:
{noformat}
"DiskHealthMonitor-Timer" #132 daemon prio=5 os_prio=0 tid=0x7f0397393000 
nid=0x26bd runnable [0x7f035e511000]
   java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.createDirectory(Native Method)
at java.io.File.mkdir(File.java:1316)
at 
org.apache.hadoop.util.DiskChecker.mkdirsWithExistsCheck(DiskChecker.java:67)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:104)
at 
org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.verifyDirUsingMkdir(DirectoryCollection.java:340)
at 
org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.testDirs(DirectoryCollection.java:312)
at 
org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.checkDirs(DirectoryCollection.java:231)
- locked <0x8065eae8> (a 
org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection)
at 
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.checkDirs(LocalDirsHandlerService.java:389)
at 
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.access$400(LocalDirsHandlerService.java:50)
at 
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService$MonitoringTimerTask.run(LocalDirsHandlerService.java:122)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
{noformat}

This disk operation could take longer time than expectation especially in high 
IO throughput case and we should have fine-grained lock for related operations 
here. 
The same issue on HDFS get raised and fixed in HDFS-7489, and we probably 
should have similar fix here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Why there are so many revert operations on trunk?

[Junping Du]

- We need to at the least force a reset of expectations w.r.t how trunk and 
small / medium / incompatible changes there are treated. We should hold off 
making a release off trunk before this gets fully discussed in the community 
and we all reach a consensus.

+1. We should hold off any release work off trunk before we reach a consensus. 
Or more and more developing work/features could be affected just like Larry 
mentioned.


- Reverts (or revert and move to a feature-branch) shouldn’t have been 
unequivocally done without dropping a note / informing everyone / building 
consensus.

Agree. To revert commits from other committers, I think we need to: 1) provide 
technical evidence/reason that is solid as rack, like: break functionality, 
tests, API compatibility, or significantly offend code convention, etc. 2) 
Making consensus with related contributors/committers based on these technical 
reasons/evidences. Unfortunately, I didn't see we ever do either thing in this 
case.


- Freaking out on -1’s and reverts - we as a community need to be less 
stigmatic about -1s / reverts.

+1. As a community, I believe we all prefer to work in a more friendly 
environment. In many cases, -1 without solid reason will frustrate people who 
are doing contributions. I think we should restraint our -1 unless it is really 
necessary.



Thanks,


Junping



From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Monday, June 06, 2016 9:36 PM
To: Andrew Wang
Cc: Junping Du; Aaron T. Myers; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: Why there are so many revert operations on trunk?

Folks,

It is truly disappointing how we are escalating situations that can be resolved 
through basic communication.

Things that shouldn’t have happened
- After a few objections were raised, commits should have simply stopped before 
restarting again but only after consensus
- Reverts (or revert and move to a feature-branch) shouldn’t have been 
unequivocally done without dropping a note / informing everyone / building 
consensus. And no, not even a release-manager gets this free pass. Not on 
branch-2, not on trunk, not anywhere.
- Freaking out on -1’s and reverts - we as a community need to be less 
stigmatic about -1s / reverts.

Trunk releases:
This is the other important bit about huge difference of expectations between 
the two sides w.r.t trunk and branching. Till now, we’ve never made releases 
out of trunk. So in-progress features that people deemed to not need a feature 
branch could go into trunk without much trouble. Given that we are now making 
releases off trunk, I can see (a) the RM saying "no, don’t put in-progress 
stuff and (b) the contributors saying “no we don’t want the overhead of a 
branch”. I’ve raised related topics (but only focusing on incompatible changes) 
before - http://markmail.org/message/m6x73t6srlchywsn - but we never decided 
anything.

We need to at the least force a reset of expectations w.r.t how trunk and small 
/ medium / incompatible changes there are treated. We should hold off making a 
release off trunk before this gets fully discussed in the community and we all 
reach a consensus.

* Without a user API, there's no way for people to use it, so not much
advantage to having it in a release

Since the code is separate and probably won't break any existing code, I
won't -1 if you want to include this in a release without a user API, but
again, I question the utility of including code that can't be used.

Clearly, there are two sides to this argument. One side claims the absence of 
user-facing public / stable APIs, and that for all purposes this is dead-code 
for everyone other than the few early adopters who want to experiment with it. 
The other argument is to not put this code before a user API. Again, I’d 
discuss with fellow community members before making what the other side 
perceives as unacceptable moves.

>From 2.8.0 perspective, it shouldn’t have landed there in the first place - I 
>have been pushing for a release for a while with help only from a few members 
>of the community. But if you say that it has no material impact on the user 
>story, having a by-default switched-off feature that *doesn’t* destabilize the 
>core release, I’d be willing to let it pass.

+Vinod

Re: Why there are so many revert operations on trunk?

[Junping Du]

Thanks Aaron for pointing it out. I didn't see any consensus on HDFS-9924 so I 
think we should bring it here with broader audiences for more discussions.

I saw several very bad practices here:

1. committer (no need to say who) revert all commits from trunk without making 
consensus with all related contributors/committers.

2. Someone's comments on feature branch are very misleading... If I didn't 
remember wrong, feature development doesn't have to go through feature branch 
which is just an optional process. This creative process of feature branch and 
branch committer - I believe the intention is trying to accelerate features 
development but not to slow them down.

3. Someone (again, no need to say who) seems to claim himself as RM for trunk. 
I don't think we need any RM for trunk. Even for RM of 3.0.0-alpha, I think we 
need someone else who demonstrates he/she is more responsible, work hardly and 
carefully and open communication with all community. Only through this, the 
success of Hadoop in age of 3.0 are guranteed.


Thanks,


Junping



From: Aaron T. Myers <a...@cloudera.com>
Sent: Monday, June 06, 2016 4:46 PM
To: Junping Du
Cc: Andrew Wang; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Why there are so many revert operations on trunk?

Junping,

All of this is being discussed on HDFS-9924. Suggest you follow the 
conversation there.

--
Aaron T. Myers
Software Engineer, Cloudera

On Mon, Jun 6, 2016 at 7:20 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi Andrew,

 I just noticed you revert 8 commits on trunk last Friday:

HADOOP-13226

HDFS-10430

HDFS-10431

HDFS-10390

HADOOP-13168

HDFS-10390

HADOOP-13168

HDFS-10346

HADOOP-12957

HDFS-10224

   And I didn't see you have any comments on JIRA or email discussion before 
you did this. I don't think we are legally allowed to do this even as 
committer/PMC member. Can you explain what's your intention to do this?

   BTW, thanks Nicolas to revert all these "illegal" revert operations.



Thanks,


Junping

Why there are so many revert operations on trunk?

[Junping Du]

Hi Andrew,

 I just noticed you revert 8 commits on trunk last Friday:

HADOOP-13226

HDFS-10430

HDFS-10431

HDFS-10390

HADOOP-13168

HDFS-10390

HADOOP-13168

HDFS-10346

HADOOP-12957

HDFS-10224

   And I didn't see you have any comments on JIRA or email discussion before 
you did this. I don't think we are legally allowed to do this even as 
committer/PMC member. Can you explain what's your intention to do this?

   BTW, thanks Nicolas to revert all these "illegal" revert operations.



Thanks,


Junping

Re: Different JIRA permissions for HADOOP and HDFS

[Junping Du]

Someone fix the permission issue so that Administrator, committer and reporter 
can edit the issue now.

Sangjin, it sounds like you were not in JIRA's committer list before and I just 
add you into committer roles for 4 projects. Hope it works for you now.​


Thanks,


Junping


From: sjl...@gmail.com <sjl...@gmail.com> on behalf of Sangjin Lee 
<sj...@apache.org>
Sent: Monday, May 16, 2016 11:43 PM
To: Zhihai Xu
Cc: Junping Du; Arun Suresh; Zheng, Kai; Andrew Wang; 
common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Different JIRA permissions for HADOOP and HDFS

I also find myself unable to edit most of the JIRA fields, and that is across 
projects (HADOOP, YARN, MAPREDUCE, and HDFS). Commenting and the workflow 
buttons seem to work, however.

On Mon, May 16, 2016 at 8:14 AM, Zhihai Xu 
<zhi...@uber.com<mailto:zhi...@uber.com>> wrote:
Great, Thanks Junping! Yes, the JIRA assignment works for me now.

zhihai

On Mon, May 16, 2016 at 5:29 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:

> Zhihai, I just set you with committer permissions on MAPREDUCE JIRA. Would
> you try if the JIRA assignment works now? I cannot help on Hive project. It
> is better to ask hive project community for help.
> For Arun's problem. from my check, the Edit permission on JIRA only
> authorized to Administrator only. I don't know if this setting is by
> intention but it was not like this previously.
> Can someone who make the change to clarify why we need this change or
> revert to whatever it used to be?
>
> Thanks,
>
> Junping
> 
> From: Arun Suresh <asur...@apache.org<mailto:asur...@apache.org>>
> Sent: Monday, May 16, 2016 9:42 AM
> To: Zhihai Xu
> Cc: Zheng, Kai; Andrew Wang; 
> common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>;
> yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>
> Subject: Re: Different JIRA permissions for HADOOP and HDFS
>
> Not sure if this is related.. but It also looks like I am now no longer
> allowed to modify description and headline of JIRAs anymore..
> Would appreciate greatly if someone can help revert this !
>
> Cheers
> -Arun
>
> On Mon, May 16, 2016 at 1:21 AM, Zhihai Xu 
> <zhi...@uber.com<mailto:zhi...@uber.com>> wrote:
>
> > Currently I also have permission issue to access the JIRA. I can't assign
> > the JIRA(I created) to myself. For example,
> > https://issues.apache.org/jira/browse/MAPREDUCE-6696 and
> > https://issues.apache.org/jira/browse/HIVE-13760. I can't find the
> button
> > to assign the JIRA to myself.
> > I don't have this issue two three weeks ago. Did anything change
> recently?
> > Can anyone help me solve this issue?
> >
> > thanks
> > zhihai
> >
> >
> >
> >
> > On Mon, May 16, 2016 at 12:22 AM, Zheng, Kai 
> > <kai.zh...@intel.com<mailto:kai.zh...@intel.com>>
> wrote:
> >
> > > It works for me now, thanks Andrew!
> > >
> > > Regards,
> > > Kai
> > >
> > > -Original Message-
> > > From: Andrew Wang 
> > > [mailto:andrew.w...@cloudera.com<mailto:andrew.w...@cloudera.com>]
> > > Sent: Monday, May 16, 2016 12:14 AM
> > > To: Zheng, Kai <kai.zh...@intel.com<mailto:kai.zh...@intel.com>>
> > > Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>
> > > Subject: Re: Different JIRA permissions for HADOOP and HDFS
> > >
> > > I just gave you committer permissions on JIRA, try now?
> > >
> > > On Mon, May 16, 2016 at 12:03 AM, Zheng, Kai 
> > > <kai.zh...@intel.com<mailto:kai.zh...@intel.com>>
> > wrote:
> > >
> > > > I just ran into the bad situation that I committed HDFS-8449 but
> can't
> > > > resolve the issue due to lacking the required permission to me. Am
> not
> > > > sure if it's caused by my setup or environment change (temporally
> > > > working in a new time zone). Would anyone help resolve the issue for
> > > > me to avoid bad state? Thanks!
> > > >
> > > > -Original Message-
> > > > From: Zheng, Kai 
> > > > [mailto:kai.zh...@intel.com<mailto:kai.zh...@intel.com>]
> > > > Sent: Sunday, May 15, 2016 3:20 PM
> > > > To: Allen Wittenauer <allenwittena...@yahoo.com.INVALID>
> > > > Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>
> > > > Subject: RE: Different JIRA permissions for HADOOP and HDFS
> > > &

Re: Different JIRA permissions for HADOOP and HDFS

[Junping Du]

Zhihai, I just set you with committer permissions on MAPREDUCE JIRA. Would you 
try if the JIRA assignment works now? I cannot help on Hive project. It is 
better to ask hive project community for help.
For Arun's problem. from my check, the Edit permission on JIRA only authorized 
to Administrator only. I don't know if this setting is by intention but it was 
not like this previously. 
Can someone who make the change to clarify why we need this change or revert to 
whatever it used to be?

Thanks,

Junping

From: Arun Suresh 
Sent: Monday, May 16, 2016 9:42 AM
To: Zhihai Xu
Cc: Zheng, Kai; Andrew Wang; common-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: Different JIRA permissions for HADOOP and HDFS

Not sure if this is related.. but It also looks like I am now no longer
allowed to modify description and headline of JIRAs anymore..
Would appreciate greatly if someone can help revert this !

Cheers
-Arun

On Mon, May 16, 2016 at 1:21 AM, Zhihai Xu  wrote:

> Currently I also have permission issue to access the JIRA. I can't assign
> the JIRA(I created) to myself. For example,
> https://issues.apache.org/jira/browse/MAPREDUCE-6696 and
> https://issues.apache.org/jira/browse/HIVE-13760. I can't find the button
> to assign the JIRA to myself.
> I don't have this issue two three weeks ago. Did anything change recently?
> Can anyone help me solve this issue?
>
> thanks
> zhihai
>
>
>
>
> On Mon, May 16, 2016 at 12:22 AM, Zheng, Kai  wrote:
>
> > It works for me now, thanks Andrew!
> >
> > Regards,
> > Kai
> >
> > -Original Message-
> > From: Andrew Wang [mailto:andrew.w...@cloudera.com]
> > Sent: Monday, May 16, 2016 12:14 AM
> > To: Zheng, Kai 
> > Cc: common-...@hadoop.apache.org
> > Subject: Re: Different JIRA permissions for HADOOP and HDFS
> >
> > I just gave you committer permissions on JIRA, try now?
> >
> > On Mon, May 16, 2016 at 12:03 AM, Zheng, Kai 
> wrote:
> >
> > > I just ran into the bad situation that I committed HDFS-8449 but can't
> > > resolve the issue due to lacking the required permission to me. Am not
> > > sure if it's caused by my setup or environment change (temporally
> > > working in a new time zone). Would anyone help resolve the issue for
> > > me to avoid bad state? Thanks!
> > >
> > > -Original Message-
> > > From: Zheng, Kai [mailto:kai.zh...@intel.com]
> > > Sent: Sunday, May 15, 2016 3:20 PM
> > > To: Allen Wittenauer 
> > > Cc: common-...@hadoop.apache.org
> > > Subject: RE: Different JIRA permissions for HADOOP and HDFS
> > >
> > > Thanks Allen for illustrating this in details. I understand. The left
> > > question is, is it intended only JIRA owner (not sure about admin
> > > users) can do the operations like updating a patch?
> > >
> > > Regards,
> > > Kai
> > >
> > > -Original Message-
> > > From: Allen Wittenauer [mailto:allenwittena...@yahoo.com.INVALID]
> > > Sent: Saturday, May 14, 2016 9:38 AM
> > > To: Zheng, Kai 
> > > Cc: common-...@hadoop.apache.org
> > > Subject: Re: Different JIRA permissions for HADOOP and HDFS
> > >
> > >
> > > > On May 14, 2016, at 7:07 AM, Zheng, Kai  wrote:
> > > >
> > > > Hi,
> > > >
> > > > Noticed this difference but not sure if it’s intended. YARN is
> > > > similar
> > > with HDFS. It’s not convenient. Any clarifying?
> > >
> > >
> > > Under JIRA, different projects (e.g., HADOOP, YARN, MAPREDUCE,
> > > HDFS, YETUS, HBASE, ACCUMULO, etc) may have different settings.  At
> > > one point in time, all of the Hadoop subprojects were under one JIRA
> > > project (HADOOP). But then a bunch of folks decided they didn’t want
> > > to see the other sub projects issues so they split them up…. and thus
> > > setting the stage for duplicate code and operational divergence in the
> > source.
> > >
> > > Since people don’t realize or care that they are separate,
> > > people will file INFRA tickets or whatever to change “their project”
> > > and not the rest. This leads to the JIRA projects also diverging…
> > > which ultimately drives those of us who actually look at the project as
> > a whole bonkers.
> > > -
> > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>