Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Junping Du]

Thanks Andrew for pushing new RC for 3.0.0. I was out last week, just get 
chance to validate new RC now.

Basically, I found two critical issues with the same rolling upgrade scenario 
as where HADOOP-15059 get found previously:
HDFS-12920, we changed value format for some hdfs configurations that old 
version MR client doesn't understand when fetching these configurations. Some 
quick workarounds are to add old value (without time unit) in hdfs-site.xml to 
override new default values but will generate many annoying warnings. I 
provided my fix suggestions on the JIRA already for more discussion.
The other one is YARN-7646. After we workaround HDFS-12920, will hit the issue 
that old version MR AppMaster cannot communicate with new version of YARN RM - 
could be related to resource profile changes from YARN side but root cause are 
still in investigation.

The first issue may not belong to a blocker given we can workaround this 
without code change. I am not sure if we can workaround 2nd issue so far. If 
not, we may have to fix this or compromise with withdrawing support of rolling 
upgrade or calling it a stable release.


Thanks,

Junping


From: Robert Kanter 
Sent: Tuesday, December 12, 2017 3:10 PM
To: Arun Suresh
Cc: Andrew Wang; Lei Xu; Wei-Chiu Chuang; Ajay Kumar; Xiao Chen; Aaron T. 
Myers; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

+1 (binding)

+ Downloaded the binary release
+ Deployed on a 3 node cluster on CentOS 7.3
+ Ran some MR jobs, clicked around the UI, etc
+ Ran some CLI commands (yarn logs, etc)

Good job everyone on Hadoop 3!


- Robert

On Tue, Dec 12, 2017 at 1:56 PM, Arun Suresh  wrote:

> +1 (binding)
>
> - Verified signatures of the source tarball.
> - built from source - using the docker build environment.
> - set up a pseudo-distributed test cluster.
> - ran basic HDFS commands
> - ran some basic MR jobs
>
> Cheers
> -Arun
>
> On Tue, Dec 12, 2017 at 1:52 PM, Andrew Wang 
> wrote:
>
> > Hi everyone,
> >
> > As a reminder, this vote closes tomorrow at 12:31pm, so please give it a
> > whack if you have time. There are already enough binding +1s to pass this
> > vote, but it'd be great to get additional validation.
> >
> > Thanks to everyone who's voted thus far!
> >
> > Best,
> > Andrew
> >
> >
> >
> > On Tue, Dec 12, 2017 at 11:08 AM, Lei Xu  wrote:
> >
> > > +1 (binding)
> > >
> > > * Verified src tarball and bin tarball, verified md5 of each.
> > > * Build source with -Pdist,native
> > > * Started a pseudo cluster
> > > * Run ec -listPolicies / -getPolicy / -setPolicy on /  , and run hdfs
> > > dfs put/get/cat on "/" with XOR-2-1 policy.
> > >
> > > Thanks Andrew for this great effort!
> > >
> > > Best,
> > >
> > >
> > > On Tue, Dec 12, 2017 at 9:55 AM, Andrew Wang  >
> > > wrote:
> > > > Hi Wei-Chiu,
> > > >
> > > > The patchprocess directory is left over from the create-release
> > process,
> > > > and it looks empty to me. We should still file a create-release JIRA
> to
> > > fix
> > > > this, but I think this is not a blocker. Would you agree?
> > > >
> > > > Best,
> > > > Andrew
> > > >
> > > > On Tue, Dec 12, 2017 at 9:44 AM, Wei-Chiu Chuang <
> weic...@cloudera.com
> > >
> > > > wrote:
> > > >
> > > >> Hi Andrew, thanks the tremendous effort.
> > > >> I found an empty "patchprocess" directory in the source tarball,
> that
> > is
> > > >> not there if you clone from github. Any chance you might have some
> > > leftover
> > > >> trash when you made the tarball?
> > > >> Not wanting to nitpicking, but you might want to double check so we
> > > don't
> > > >> ship anything private to you in public :)
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Dec 12, 2017 at 7:48 AM, Ajay Kumar <
> > ajay.ku...@hortonworks.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> +1 (non-binding)
> > > >>> Thanks for driving this, Andrew Wang!!
> > > >>>
> > > >>> - downloaded the src tarball and verified md5 checksum
> > > >>> - built from source with jdk 1.8.0_111-b14
> > > >>> - brought up a pseudo distributed cluster
> > > >>> - did basic file system operations (mkdir, list, put, cat) and
> > > >>> confirmed that everything was working
> > > >>> - Run word count, pi and DFSIOTest
> > > >>> - run hdfs and yarn, confirmed that the NN, RM web UI worked
> > > >>>
> > > >>> Cheers,
> > > >>> Ajay
> > > >>>
> > > >>> On 12/11/17, 9:35 PM, "Xiao Chen"  wrote:
> > > >>>
> > > >>> +1 (binding)
> > > >>>
> > > >>> - downloaded src tarball, verified md5
> > > >>> - built from source with jdk1.8.0_112
> > > >>> - started a pseudo cluster with hdfs and kms
> > > >>> - sanity checked encryption related operations working
> > > >>> - sanity checked webui and logs.
> > > >>>
> > > >>>

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Junping Du]

Hi Konstantin,

 Thanks for verification and comments. I was verifying your example below 
but found it is actually matched:


jduMBP:hadoop-2.8.3 jdu$ md5 ~/Downloads/hadoop-2.8.3-src.tar.gz
MD5 (/Users/jdu/Downloads/hadoop-2.8.3-src.tar.gz) = 
e53d04477b85e8b58ac0a26468f04736

What's your md5 checksum for given source tar ball?


Thanks,


Junping



From: Konstantin Shvachko <shv.had...@gmail.com>
Sent: Saturday, December 9, 2017 11:06 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

Hey Junping,

Could you pls upload mds relative to the tar.gz etc. files rather than their 
full path
/build/source/target/artifacts/hadoop-2.8.3-src.tar.gz:
   MD5 = E5 3D 04 47 7B 85 E8 B5  8A C0 A2 64 68 F0 47 36

Otherwise mds don't match for me.

Thanks,
--Konstantin

On Tue, Dec 5, 2017 at 1:58 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi all,
 I've created the first release candidate (RC0) for Apache Hadoop 2.8.3. 
This is our next maint release to follow up 2.8.2. It includes 79 important 
fixes and improvements.

  The RC artifacts are available at: 
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0

  The RC tag in git is: release-2.8.3-RC0

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1072

  Please try the release and vote; the vote will run for the usual 5 
working days, ending on 12/12/2017 PST time.

Thanks,

Junping

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Junping Du]

Thanks Eric for verification!

A kindly reminder: The original due date for 2.8.3 voting is tomorrow but I 
only receive 1 binding vote so far - we have 76 PMCs and 127 Committers!

I can understand the whole community are busy with 3 release RC voting (2.7.5, 
2.8.3 and 3.0.0) and may be it is necessary to extend the voting period for a 
few more days. But please try as much as possible to verify our release bits. 
Thanks!


Thanks,


Junping?



From: Eric Payne <eric.payne1...@yahoo.com>
Sent: Monday, December 11, 2017 1:51 PM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

Thanks Junping for the hard work on this release.

+1 (binding)

On a 6 node pseudo cluster (4 NMs), I performed the following manual tests:

- Built and installed from source

- Successfully ran a stream job

- Verified that user weights are honored by assigning the appropriate amount of 
resources to the weighted users.

- Ensured that FiarOrderingPolicy and FifoOrderingPolicy worked in the Capacity 
Scheduler as expected

- Applications with higher priorities are assigned containers as expected in 
the FifoOrderingPolicy of the Capacity Scheduler until the user reaches its 
user resource limit.

Eric Payne



From: Junping Du <j...@hortonworks.com>
To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>
Sent: Tuesday, December 5, 2017 3:58 AM
Subject: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

Hi all,
I've created the first release candidate (RC0) for Apache Hadoop 2.8.3. 
This is our next maint release to follow up 2.8.2. It includes 79 important 
fixes and improvements.

  The RC artifacts are available at: 
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0

  The RC tag in git is: release-2.8.3-RC0

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1072

  Please try the release and vote; the vote will run for the usual 5 
working days, ending on 12/12/2017 PST time.

Thanks,

Junping

[VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Junping Du]

Hi all,
 I've created the first release candidate (RC0) for Apache Hadoop 2.8.3. 
This is our next maint release to follow up 2.8.2. It includes 79 important 
fixes and improvements.

  The RC artifacts are available at: 
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0

  The RC tag in git is: release-2.8.3-RC0

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1072

  Please try the release and vote; the vote will run for the usual 5 
working days, ending on 12/12/2017 PST time.

Thanks,

Junping

Re: Apache Hadoop 2.8.3 Release Plan

[Junping Du]

Thanks Konstantin and Weiwei for calling attention. These two HDFS issues looks 
to be important to be fixed which are on my radar now.

I will hold on RC cut until we figure them out.


Thanks,


Junping



From: Weiwei Yang <cheersy...@hotmail.com>
Sent: Tuesday, November 21, 2017 7:45 PM
To: Junping Du; Konstantin Shvachko
Cc: Wangda Tan; Andrew Wang; Zheng, Kai; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

Agree with Konstantin. This two issues has been opened for a while but could 
not reach a consensus on the fix, hope this gets enough attention from the 
community and get them resolved.

Thanks

--
Weiwei

On 22 Nov 2017, 11:18 AM +0800, Konstantin Shvachko <shv.had...@gmail.com>, 
wrote:
I would consider these two blockers for 2.8.3 as they crash NN:

https://issues.apache.org/jira/browse/HDFS-12638
https://issues.apache.org/jira/browse/HDFS-12832

Thanks,
--Konstantin

On Tue, Nov 21, 2017 at 11:16 AM, Junping Du <j...@hortonworks.com> wrote:

Thanks Andrew and Wangda for comments!

To me, an improvement with 17 patches is not a big problem as this is
self-contained and I didn't see a single line of delete/update on existing
code - well, arguably, patches with only adding code can also have big
impact but not likely the case here.

While the dependency discussions on HADOOP-14964 are still going on, I
will leave the decision to JIRA discussion based on which approach we will
choose(shaded?) and impact. If we cannot make consensus in short term,
probably we have to miss this in 2.8.3 release.


Okay. Last call for blocker/critical fixes landing on branch-2.8.3. RC0
will get cut-off shortly.



Thanks,


Junping



From: Wangda Tan <wheele...@gmail.com
Sent: Tuesday, November 21, 2017 10:52 AM
To: Andrew Wang
Cc: Junping Du; Zheng, Kai; common-...@hadoop.apache.org;
hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org;
yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

Thanks Junping for driving this.

For the bug fix vs. improvement, it is actually very hard to define,
improvement could be self-contained and useful, bug fix could be dangerous
in some cases. To me, If an improvement fixed some existing use case, and
the fix is self-contained. I will be open to bring such fix to maintenance
release. For example, in 2.8.2, we back ported CapacityScheduler intra
queue preemption https://issues.apache.org/jira/browse/YARN-2113. It is a
big change in terms of patch size, but since it fixes broken use case
(balance user usage under Capacity Scheduler leaf queue), we backported it
to 2.8.2 after thorough tests and validations by Yahoo.

I'm not quite familiar with HADOOP-14964, I will leave the decision to
committers who know more about the field.

Just my 2 cents.

Regards,
Wangda


On Tue, Nov 21, 2017 at 10:21 AM, Andrew Wang <andrew.w...@cloudera.com<
mailto:andrew.w...@cloudera.com>> wrote:
The Aliyun OSS code isn't a small improvement. If you look at Sammi's
comment
<https://issues.apache.org/jira/browse/HADOOP-14964?
focusedCommentId=16247085=com.atlassian.jira.
plugin.system.issuetabpanels:comment-tabpanel#comment-16247085>,
it's
a 17 patch series that is being backported in one shot. What we're talking
about is equivalent to merging a feature branch in a maintenance release. I
see that Kai and Chris are having a discussion about the dependency
changes, which indicates this is not a zero-risk change either. We really
should not be changing dependency versions in a maintenance unless it's
because of a bug.

It's unfortunate from a timing perspective that this missed 2.9.0, but I
still think it should wait for the next minor. Merging a feature into a
maintenance release sets the wrong precedent.

Best,
Andrew

On Tue, Nov 21, 2017 at 1:08 AM, Junping Du <j...@hortonworks.com<mailto:jd
u...@hortonworks.com>> wrote:

Thanks Kai for calling out this feature/improvement for attention and
Andrew for comments.


While I agree that maintenance release should focus on important bug fix
only, I doubt we have strict rules to disallow any features/improvements
to
land on maint release especially when those are small footprint or low
impact on existing code/features. In practice, we indeed has 77 new
features/improvements in latest 2.7.3 and 2.7.4 release.


Back to HADOOP-14964, I did a quick check and it looks like case here
belongs to self-contained improvement that has very low impact on
existing
code base, so I am OK with the improvement get landed on branch-2.8 in
case
it is well reviewed and tested.


However, as RM of branch-2.8, I have two concerns to accept it in our
2.8.3 release:

1. Timing - as I mentioned in beginning, the main purpose of 2.8.3 are
for
several critical bug fixes and we should target to release it very soon -
my cu

Re: [DISCUSS] A final minor release off branch-2?

[Junping Du]

Hi Andrew,

bq. Source and binary compatibility are not required for 3.0.0. It's a new 
major release, and there are known, documented incompatibilities in this regard.

Technically, it is true. However, in practically, we should retain 
compatibility as much as we can. Otherwise, we could break downstream projects, 
third-party libraries and existing users applications unintentionally. A quick 
example here is a blocker issue I just reported in HADOOP-15059 which break old 
(2.x) MR application with 3.0 deployment - due to token format incompatible 
issue.


bq. To follow up on my earlier email, I don't think there's need for a bridge 
release given that we've successfully tested rolling upgrade from 2.x to 3.0.0.​

Did we find the same issue as HADOOP-15059? If so, just curious on what rolling 
upgrade means here - IMO, upgrade with breaking running applications shouldn't 
be recognized as "rolling". Do I miss anything?



Thanks,


Junping



From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Wednesday, November 15, 2017 10:34 AM
To: Junping Du
Cc: Wangda Tan; Steve Loughran; Vinod Kumar Vavilapalli; Kai Zheng; Arun 
Suresh; common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Hdfs-dev; 
mapreduce-dev@hadoop.apache.org
Subject: Re: [DISCUSS] A final minor release off branch-2?

Hi Junping,

On Wed, Nov 15, 2017 at 1:37 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Thanks Vinod to bring up this discussion, which is just in time.

I agree with most responses that option C is not a good choice as our community 
bandwidth is precious and we should focus on very limited mainstream branches 
to develop, test and deployment. Of course, we should still follow Apache way 
to allow any interested committer for rolling up his/her own release given 
specific requirement over the mainstream releases.

I am not biased on option A or B (I will discuss this later), but I think a 
bridge release for upgrading to and back from 3.x is very necessary.
The reasons are obviously:
1. Given lesson learned from previous experience of migration from 1.x to 2.x, 
no matter how careful we tend to be, there is still chance that some level of 
compatibility (source, binary, configuration, etc.) get broken for the 
migration to new major release. Some of these incompatibilities can only be 
identified in runtime after GA release with widely deployed in production 
cluster - we have tons of downstream projects and numerous configurations and 
we cannot cover them all from in-house deployment and test.

Source and binary compatibility are not required for 3.0.0. It's a new major 
release, and there are known, documented incompatibilities in this regard.

That said, we've done far, far more in this regard compared to previous major 
or minor releases. We've compiled all of CDH against Hadoop 3 and run our suite 
of system tests for the platform. We've been testing in this way since 
3.0.0-alpha1 and found and fixed plenty of source and binary compatibility 
issues during the alpha and beta process. Many of these fixes trickled down 
into 2.8 and 2.9.

2. From recent classpath isolation work, I was surprised to find out that many 
of our downstream projects (HBase, Tez, etc.) are still consuming many 
non-public, server side APIs of Hadoop, not saying the projects/products 
outside of hadoop ecosystem. Our API compatibility test does not (and should 
not) cover these cases and situations. We can claim that new major release 
shouldn't be responsible for these private API changes. But given the 
possibility of breaking existing applications in some way, users could be very 
hesitated to migrate to 3.x release if there is no safe solution to roll back.

This is true for 2.x releases as well. Similar to the previous answer, we've 
compiled all of CDH against Hadoop 3, providing a much higher level of 
assurance even compared to 2.x releases.

3. Beside incompatibilities, there is also possible to have performance 
regressions (lower throughput, higher latency, slower job running, bigger 
memory footprint or even memory leaking, etc.) for new hadoop releases. While 
the performance impact of migration (if any) could be neglectable to some 
users, other users could be very sensitive and wish to roll back if it happens 
on their production cluster.

Yes, bugs exist. I won't claim that 3.0.0 is bug-free. All new releases can 
potentially introduce new bugs.

However, I don't think rollback is the solution. In my experience, users rarely 
rollback since it's so disruptive and causes data loss. It's much more common 
that they patch and upgrade. With that in mind, I'd rather we spend our effort 
on making 3.0.x high-quality vs. making it easier to rollback.

The root of my concern in announcing a "bridge release" is that it discourages 
users from upgrading to 3.0.0 until a bridge release is out. I strongly believe 
the level of quality provid

Re:

[Junping Du]

Just filed HADOOP-15059 to track this.


Thanks,


Junping



From: Junping Du
Sent: Tuesday, November 21, 2017 1:09 PM
To: Vinod Kumar Vavilapalli; Allen Wittenauer
Cc: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

Another issue I just found is we have token format compatibility issue between 
2.x and 3.0. I tried to run a simple MR job on 3.0 RC0 against with 2.9.0 
tarball which is failed. This incompatibility change should also break other 
applications which will break rolling upgrade.
I think it is another blocker for 3.0.0 release.

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Tuesday, November 21, 2017 12:59 PM
To: Allen Wittenauer
Cc: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

>>> - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>>
>>
>> The yarn-ui profile has always been off by default, AFAIK. It's documented
>> to turn it on in BUILDING.txt for release builds, and we do it in
>> create-release.
>>
>> IMO not a blocker. I think it's also more of a dev question (do we want to
>> do this on every YARN build?) than a release one.
>
>   -1 on making yarn-ui always build.
>
>   For what is effectively an optional component (the old UI is still 
> there), it's heavy dependency requirements make it a special burden outside 
> of the Docker container.


Thanks for the background on this.

I got schooled offline too about the heaviness of the dependencies.


> If it can be changed such that it either always downloads the necessary bits 
> (regardless of the OS/chipset!) and/or doesn't kill the maven build if those 
> bits can't be found  (i.e., truly optional), then I'd be less opposed.  (and, 
> actually, quite pleased because then the docker image build would be 
> significantly faster.)


This is a good idea (to the extend I understand your proposal). Long term, we'd 
like YARN UI2 to replace current UI, so it shouldn't be optionally build and 
hence the build should be fast. Let me track this separately as a non-blocker.

Thanks
+Vinod
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Junping Du]

Another issue I just found is we have token format compatibility issue between 
2.x and 3.0. I tried to run a simple MR job on 3.0 RC0 against with 2.9.0 
tarball which is failed. This incompatibility change should also break other 
applications which will break rolling upgrade.
I think it is another blocker for 3.0.0 release.

Thanks,

Junping

From: Vinod Kumar Vavilapalli 
Sent: Tuesday, November 21, 2017 12:59 PM
To: Allen Wittenauer
Cc: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

>>> - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>>
>>
>> The yarn-ui profile has always been off by default, AFAIK. It's documented
>> to turn it on in BUILDING.txt for release builds, and we do it in
>> create-release.
>>
>> IMO not a blocker. I think it's also more of a dev question (do we want to
>> do this on every YARN build?) than a release one.
>
>   -1 on making yarn-ui always build.
>
>   For what is effectively an optional component (the old UI is still 
> there), it’s heavy dependency requirements make it a special burden outside 
> of the Docker container.


Thanks for the background on this.

I got schooled offline too about the heaviness of the dependencies.


> If it can be changed such that it either always downloads the necessary bits 
> (regardless of the OS/chipset!) and/or doesn’t kill the maven build if those 
> bits can’t be found  (i.e., truly optional), then I’d be less opposed.  (and, 
> actually, quite pleased because then the docker image build would be 
> significantly faster.)


This is a good idea (to the extend I understand your proposal). Long term, we'd 
like YARN UI2 to replace current UI, so it shouldn't be optionally build and 
hence the build should be fast. Let me track this separately as a non-blocker.

Thanks
+Vinod
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7012) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade

[Junping Du (JIRA)]

Junping Du created MAPREDUCE-7012:
-

 Summary: 3.0 deployment cannot work with old version MR tar ball 
which break rolling upgrade
 Key: MAPREDUCE-7012
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7012
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: distributed-cache, mrv2
Reporter: Junping Du
Priority: Blocker


I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed 
because following error:
{noformat}
2017-11-21 12:42:50,911 INFO [main] 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for 
application appattempt_1511295641738_0003_01
2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
Unable to load native-hadoop library for your platform... using builtin-java 
classes where applicable
2017-11-21 12:42:51,118 FATAL [main] 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.RuntimeException: Unable to determine current user
at 
org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
at 
org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
at 
org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
at 
org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
Caused by: java.io.IOException: Exception reading 
/tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
at 
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
at 
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
at 
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
at 
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
at 
org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
... 4 more
Caused by: java.io.IOException: Unknown version 1 in token storage.
at 
org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
at 
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
... 8 more
2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting 
with status 1: java.lang.RuntimeException: Unable to determine current user
{noformat}
I think it is due to token incompatiblity change between 2.9 and 3.0. As we 
claim "rolling upgrade" is supported in Hadoop 3, we should fix this before we 
ship 3.0 otherwise all MR running applications will get stuck during/after 
upgrade.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.3 Release Plan

[Junping Du]

Thanks Andrew and Wangda for comments!

To me, an improvement with 17 patches is not a big problem as this is 
self-contained and I didn't see a single line of delete/update on existing code 
- well, arguably, patches with only adding code can also have big impact but 
not likely the case here.

While the dependency discussions on HADOOP-14964 are still going on, I will 
leave the decision to JIRA discussion based on which approach we will 
choose(shaded?) and impact. If we cannot make consensus in short term, probably 
we have to miss this in 2.8.3 release.


Okay. Last call for blocker/critical fixes landing on branch-2.8.3. RC0 will 
get cut-off shortly.



Thanks,


Junping



From: Wangda Tan <wheele...@gmail.com>
Sent: Tuesday, November 21, 2017 10:52 AM
To: Andrew Wang
Cc: Junping Du; Zheng, Kai; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

Thanks Junping for driving this.

For the bug fix vs. improvement, it is actually very hard to define, 
improvement could be self-contained and useful, bug fix could be dangerous in 
some cases. To me, If an improvement fixed some existing use case, and the fix 
is self-contained. I will be open to bring such fix to maintenance release. For 
example, in 2.8.2, we back ported CapacityScheduler intra queue preemption 
https://issues.apache.org/jira/browse/YARN-2113. It is a big change in terms of 
patch size, but since it fixes broken use case (balance user usage under 
Capacity Scheduler leaf queue), we backported it to 2.8.2 after thorough tests 
and validations by Yahoo.

I'm not quite familiar with HADOOP-14964, I will leave the decision to 
committers who know more about the field.

Just my 2 cents.

Regards,
Wangda


On Tue, Nov 21, 2017 at 10:21 AM, Andrew Wang 
<andrew.w...@cloudera.com<mailto:andrew.w...@cloudera.com>> wrote:
The Aliyun OSS code isn't a small improvement. If you look at Sammi's
comment
<https://issues.apache.org/jira/browse/HADOOP-14964?focusedCommentId=16247085=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16247085>,
it's
a 17 patch series that is being backported in one shot. What we're talking
about is equivalent to merging a feature branch in a maintenance release. I
see that Kai and Chris are having a discussion about the dependency
changes, which indicates this is not a zero-risk change either. We really
should not be changing dependency versions in a maintenance unless it's
because of a bug.

It's unfortunate from a timing perspective that this missed 2.9.0, but I
still think it should wait for the next minor. Merging a feature into a
maintenance release sets the wrong precedent.

Best,
Andrew

On Tue, Nov 21, 2017 at 1:08 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:

> Thanks Kai for calling out this feature/improvement for attention and
> Andrew for comments.
>
>
> While I agree that maintenance release should focus on important bug fix
> only, I doubt we have strict rules to disallow any features/improvements to
> land on maint release especially when those are small footprint or low
> impact on existing code/features. In practice, we indeed has 77 new
> features/improvements in latest 2.7.3 and 2.7.4 release.
>
>
> Back to HADOOP-14964, I did a quick check and it looks like case here
> belongs to self-contained improvement that has very low impact on existing
> code base, so I am OK with the improvement get landed on branch-2.8 in case
> it is well reviewed and tested.
>
>
> However, as RM of branch-2.8, I have two concerns to accept it in our
> 2.8.3 release:
>
> 1. Timing - as I mentioned in beginning, the main purpose of 2.8.3 are for
> several critical bug fixes and we should target to release it very soon -
> my current plan is to cut RC out within this week inline with waiting
> for 3.0.0 vote closing. Can this improvement be well tested against
> branch-2.8.3 within this strictly timeline? It seems a bit rush unless we
> have strong commitment on test plan and activities in such a tight time.
>
>
> 2. Upgrading - I haven't heard we settle down the plan of releasing this
> feature in 2.9.1 release - though I saw some discussions are going on
> at HADOOP-14964. Assume 2.8.3 is released ahead of 2.9.1 and it includes
> this improvement, then users consuming this feature/improvement have no 2.9
> release to upgrade or forcefully upgrade with regression. We may need a
> better upgrade story here.
>
>
> Pls let me know what you think. Thanks!
>
>
>
> Thanks,
>
>
> Junping
>
>
> ----------
> *From:* Andrew Wang 
> <andrew.w...@cloudera.com<mailto:andrew.w...@cloudera.com>>
> *Sent:* Monday, November 20, 20

Re: Apache Hadoop 2.8.3 Release Plan

[Junping Du]

Thanks Kai for calling out this feature/improvement for attention and Andrew 
for comments.


While I agree that maintenance release should focus on important bug fix only, 
I doubt we have strict rules to disallow any features/improvements to land on 
maint release especially when those are small footprint or low impact on 
existing code/features. In practice, we indeed has 77 new features/improvements 
in latest 2.7.3 and 2.7.4 release.


Back to HADOOP-14964, I did a quick check and it looks like case here belongs 
to self-contained improvement that has very low impact on existing code base, 
so I am OK with the improvement get landed on branch-2.8 in case it is well 
reviewed and tested.


However, as RM of branch-2.8, I have two concerns to accept it in our 2.8.3 
release:

1. Timing - as I mentioned in beginning, the main purpose of 2.8.3 are for 
several critical bug fixes and we should target to release it very soon - my 
current plan is to cut RC out within this week inline with waiting for 3.0.0 
vote closing. Can this improvement be well tested against branch-2.8.3 within 
this strictly timeline? It seems a bit rush unless we have strong commitment on 
test plan and activities in such a tight time.


2. Upgrading - I haven't heard we settle down the plan of releasing this 
feature in 2.9.1 release - though I saw some discussions are going on at 
HADOOP-14964. Assume 2.8.3 is released ahead of 2.9.1 and it includes this 
improvement, then users consuming this feature/improvement have no 2.9 release 
to upgrade or forcefully upgrade with regression. We may need a better upgrade 
story here.


Pls let me know what you think. Thanks!



Thanks,


Junping



From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Monday, November 20, 2017 10:22 PM
To: Zheng, Kai
Cc: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

I'm against including new features in maintenance releases, since they're meant 
to be bug-fix only.

If we're struggling with being able to deliver new features in a safe and 
timely fashion, let's try to address that, not overload the meaning of 
"maintenance release".

Best,
Andrew

On Mon, Nov 20, 2017 at 5:20 PM, Zheng, Kai 
<kai.zh...@intel.com<mailto:kai.zh...@intel.com>> wrote:
Hi Junping,

Thank you for making 2.8.2 happen and now planning the 2.8.3 release.

I have an ask, is it convenient to include the back port work for OSS connector 
module? We have some Hadoop users that wish to have it by default for 
convenience, though in the past they used it by back porting themselves. I have 
raised this and got thoughts from Chris and Steve. Looks like this is more 
wanted for 2.9 but I wanted to ask again here for broad feedback and thoughts 
by this chance. The back port patch is available for 2.8 and the one for 
branch-2 was already in. IMO, 2.8.x is promising as we can see some shift from 
2.7.x, hence it's worth more important features and efforts. How would you 
think? Thanks!

https://issues.apache.org/jira/browse/HADOOP-14964

Regards,
Kai

-Original Message-
From: Junping Du [mailto:j...@hortonworks.com<mailto:j...@hortonworks.com>]
Sent: Tuesday, November 14, 2017 9:02 AM
To: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>
Subject: Apache Hadoop 2.8.3 Release Plan

Hi,
We have several important fixes get landed on branch-2.8 and I would like 
to cut off branch-2.8.3 now to start 2.8.3 release work.
So far, I don't see any pending blockers on 2.8.3, so my current plan is to 
cut off 1st RC of 2.8.3 in next several days:
 -  For all coming commits to land on branch-2.8, please mark the fix 
version as 2.8.4.
 -  If there is a really important fix for 2.8.3 and getting closed, 
please notify me ahead before landing it on branch-2.8.3.
Please let me know if you have any thoughts or comments on the plan.

Thanks,

Junping

From: dujunp...@gmail.com<mailto:dujunp...@gmail.com> 
<dujunp...@gmail.com<mailto:dujunp...@gmail.com>> on behalf of ??? 
<junping...@apache.org<mailto:junping...@apache.org>>
Sent: Friday, October 27, 2017 3:33 PM
To: gene...@hadoop.apache.org<mailto:gene...@hadoop.apache.org>
Subject: [ANNOUNCE] Apache Hadoop 2.8.2 Release.

Hi all,

It gives me great pleasure to announce that the Apache Hadoop community has 
voted to release Apache Hadoop 2.8.2, which is now available for download from 
Apache mirrors[1]. For download instructions please refer to the Apache Hadoop 
Release page [2].

Apache Hadoop 2.8.2 is the firs

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Junping Du]

I did following verification work:

- build succeed from source
- verify signature
- deploy a pseudo cluster and run some simple MR jobs
- Check HDFS/YARN daemons' UI
- Verify commit history matching with JIRA's fix version (which will be 
included in site document).

Most works fine, except I found ~200 JIRA are claimed to be included in 2.9.0 
but actually not show up in commit log for following cases:
a. commits are missing in 2.9.0
b. JIRA is marked as resolved but not fixed which shouldn't have fix version
c. JIRA is umbrella which doesn't include particular commits
d. JIRA commit message is lacking of JIRA number or have wrong number
e. JIRA commit is included in a whole commit due to branch merge work, like ATS 
v2.

While c,d,e is something we have to live with, but I hope we can enhance a. and 
b. next time.


Assume document issue is not a blocker for release, I give my bind +1.


Thanks,

Junping


From: Arun Suresh 
Sent: Monday, November 13, 2017 4:10 PM
To: yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; Hadoop Common; 
Hdfs-dev
Cc: Subramaniam Krishnan
Subject: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

Hi Folks,

Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
2.8.2.

More information about the 2.9.0 release plan can be found here:
*https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
*

New RC is available at: *https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
*

The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
756ebc8394e473ac25feac05fa493f6d612e6c50.

The maven artifacts are available via repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1068/
*

We are carrying over the votes from the previous RC given that the delta is
the license fix.

Given the above - we are also going to stick with the original deadline for
the vote : ending on Friday 17th November 2017 2pm PT time.

Thanks,
-Arun/Subru


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] A final minor release off branch-2?

[Junping Du]

Thanks Vinod to bring up this discussion, which is just in time.

I agree with most responses that option C is not a good choice as our community 
bandwidth is precious and we should focus on very limited mainstream branches 
to develop, test and deployment. Of course, we should still follow Apache way 
to allow any interested committer for rolling up his/her own release given 
specific requirement over the mainstream releases.

I am not biased on option A or B (I will discuss this later), but I think a 
bridge release for upgrading to and back from 3.x is very necessary. 
The reasons are obviously:
1. Given lesson learned from previous experience of migration from 1.x to 2.x, 
no matter how careful we tend to be, there is still chance that some level of 
compatibility (source, binary, configuration, etc.) get broken for the 
migration to new major release. Some of these incompatibilities can only be 
identified in runtime after GA release with widely deployed in production 
cluster - we have tons of downstream projects and numerous configurations and 
we cannot cover them all from in-house deployment and test.

2. From recent classpath isolation work, I was surprised to find out that many 
of our downstream projects (HBase, Tez, etc.) are still consuming many 
non-public, server side APIs of Hadoop, not saying the projects/products 
outside of hadoop ecosystem. Our API compatibility test does not (and should 
not) cover these cases and situations. We can claim that new major release 
shouldn't be responsible for these private API changes. But given the 
possibility of breaking existing applications in some way, users could be very 
hesitated to migrate to 3.x release if there is no safe solution to roll back. 

3. Beside incompatibilities, there is also possible to have performance 
regressions (lower throughput, higher latency, slower job running, bigger 
memory footprint or even memory leaking, etc.) for new hadoop releases. While 
the performance impact of migration (if any) could be neglectable to some 
users, other users could be very sensitive and wish to roll back if it happens 
on their production cluster.

As Andrew mentioned in early email threads, some work has been done for 
verifying rolling upgrade from 2.x to 3.0 (just curious that which 2.x release 
is tested to upgrade from? 2.8.2 or 2.9.0 which is still in releasing?). But I 
am not aware any work we are doing now to test downgrade from 3.0 to 2.x 
(correct me if I miss any work). If users hit any of three situations I 
mentioned above then we should give them the chance to roll back if they are 
really conservative to these unexpected side-effect of upgrading. Given this, 
we should have this bridge release to cover the case for 3.0 safely roll back 
(no matter rolling or not). I am not sure it should be 2.9.x or 2.10.x for now 
(we can just call it 2.BR release) because we are not sure what exactly changes 
we should include for supporting roll back from 3.0 at this moment. We can 
defer this decision to discuss later when we have better ideas.

Summary for my two cents:
- No more feature release should happen on branch-2. 2.9 or 2.10 should be the 
last minor release (mainstream of community) on branch-2

- A bridge release is necessary for safely upgrade/downgrade to 3.x

- We can decide later to see if 2.10 is necessary when scope of the bridge 
release is more clear.


Thanks,

Junping


From: Andrew Wang 
Sent: Tuesday, November 14, 2017 2:25 PM
To: Wangda Tan
Cc: Steve Loughran; Vinod Kumar Vavilapalli; Kai Zheng; Arun Suresh; 
common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Hdfs-dev; 
mapreduce-dev@hadoop.apache.org
Subject: Re: [DISCUSS] A final minor release off branch-2?

To follow up on my earlier email, I don't think there's need for a bridge
release given that we've successfully tested rolling upgrade from 2.x to
3.0.0. I expect we'll keep making improvements to smooth over any
additional incompatibilities found, but there isn't a requirement that a
user upgrade to a bridge release before upgrading to 3.0.

Otherwise, I don't have a strong opinion about when to discontinue branch-2
releases. Historically, a release line is maintained until interest in it
wanes. If the maintainers are taking care of the backports, it's not much
work for the rest of us to vote on the RCs.

Best,
Andrew

On Mon, Nov 13, 2017 at 4:19 PM, Wangda Tan  wrote:

> Thanks Vinod for staring this,
>
> I'm also leaning towards the plan (A):
>
>
>
>
> * (A)-- Make 2.9.x the last minor release off branch-2-- Have a
> maintenance release that bridges 2.9 to 3.x-- Continue to make more
> maintenance releases on 2.8 and 2.9 as necessary*
>
> The only part I'm not sure is having a separate bridge release other than
> 3.x.
>
> For the bridge release, Steve's suggestion sounds more doable:
>
> ** 3.1+ for new features*
> ** fixes to 3.0.x &, where 

Apache Hadoop 2.8.3 Release Plan

[Junping Du]

Hi,
We have several important fixes get landed on branch-2.8 and I would like 
to cut off branch-2.8.3 now to start 2.8.3 release work. 
So far, I don't see any pending blockers on 2.8.3, so my current plan is to 
cut off 1st RC of 2.8.3 in next several days: 
 -  For all coming commits to land on branch-2.8, please mark the fix 
version as 2.8.4.
 -  If there is a really important fix for 2.8.3 and getting closed, 
please notify me ahead before landing it on branch-2.8.3.
Please let me know if you have any thoughts or comments on the plan.

Thanks,

Junping

From: dujunp...@gmail.com  on behalf of 俊平堵 

Sent: Friday, October 27, 2017 3:33 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.2 Release.

Hi all,

It gives me great pleasure to announce that the Apache Hadoop community
has
voted to release Apache Hadoop 2.8.2, which is now available for download
from
Apache mirrors[1]. For download instructions please refer to the Apache
Hadoop
Release page [2].

Apache Hadoop 2.8.2 is the first GA release of Apache Hadoop 2.8 line and
our
newest stable release for entire Apache Hadoop project. For major changes
incuded in Hadoop 2.8 line, please refer Hadoop 2.8.2 main page[3].

This release has 315 resolved issues since previous 2.8.1 release with
following
breakdown:
   - 91 in Hadoop Common
   - 99 in HDFS
   - 105 in YARN
   - 20 in MapReduce
Please read the log of CHANGES[4] and RELEASENOTES[5] for more details.

The release news is posted on the Hadoop website too, you can go to the
downloads section directly [6].

Thank you all for contributing to the Apache Hadoop release!


Cheers,

Junping


[1] http://www.apache.org/dyn/closer.cgi/hadoop/common

[2] http://hadoop.apache.org/releases.html

[3] http://hadoop.apache.org/docs/r2.8.2/index.html

[4]
http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/release/2.8.2/CHANGES.2.8.2.html

[5]
http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/release/2.8.2/RELEASENOTES.2.8.2.html

[6] http://hadoop.apache.org/releases.html#Download


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Cutting branch-2.9

[Junping Du]

Below is some findings from my recently run of JACC 
(https://builds.apache.org/job/Hadoop-2.9-JACC/12/artifact/target/compat-check/report.html)
 job against 2.9 and 2.8.2. I have discussed with Arun and Subru offline on 
jdiff report who convinced me some of items are not a big concern. Just tried 
to list here in case people in community have different voices:

 My original concerns of incompatibilities:
 1. It looks like we change from log4j to sel4j (HADOOP-12956) in 2.9 which 
is a huge incompatible change. Do we have consensus on this in community - as I 
saw HBase community are deny this kind of change in minor release? I talked 
with several other guys who seems to have different opinion.

2. Some APIs (deprecated in 2.8) get removed. My understanding is we only 
remove deprecated API in next major release (3.0) but not minor release. - Arun 
and Subru convinced me that removed 2 methods are not supposed to be used by 
other downstream projects

3. Some stable APIs get removed, like: YarnScheduler.allocate(), etc. We 
may need to add it back for compatibility of Scheduler API.

Thoughts?

Thanks,

Junping


From: Arun Suresh 
Sent: Thursday, November 2, 2017 11:55 PM
To: 俊平堵
Cc: Arun Suresh; su...@apache.org; common-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org
Subject: Re: Cutting branch-2.9

Quick update folks.

We just cut branch-2.9.0, and set the mvn version to 2.9.0
We've also updated the version on branch-2.9 to 2.9.1-SNAPSHOT.

Expect an RC in the next 24 hrs :)

Cheers
-Arun/Subru

On Tue, Oct 31, 2017 at 5:24 PM, Arun Suresh  wrote:

> Hi Junping,
>
> > In the mean time, branch-2.9 should be reserved for next 2.9 point
> release (2.9.1) and branch-2 should be reserved for next minor release
> (2.10.0 or whatever name it is). Thoughts?
> Yup, That is the current plan. We've updated mvn versions in branch-2 to
> point to 2.10.0-SNAPSHOT.
> Once we cut branch-2.9.0, we will update mvn versions on branch-2.9.0 to
> 2.9.0 and set versions on branch-2.9 to 2.9.1-SNAPSHOT (We plan to do that
> by Thursday)
>
> Cheers
> -Arun
>
> On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵  wrote:
>
>> Hi Arun/Subru,
>> Thanks for updating on 2.9.0 release progress. A quick question here:
>> are we planning to release from branch-2.9 directly?
>> I doubt this as it seems against our current branch release practice (
>> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of
>> any confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release
>> work. In the mean time, branch-2.9 should be reserved for next 2.9 point
>> release (2.9.1) and branch-2 should be reserved for next minor release
>> (2.10.0 or whatever name it is). Thoughts?
>>
>> bq. @Junping, lets move the jdiff conversation to separate thread.
>> Sure. I will reply jdiff in separated thread.
>>
>> Thanks,
>>
>> Junping
>>
>> 2017-10-31 13:44 GMT-07:00 Arun Suresh :
>>
>>> Hello folks
>>>
>>> We just cut branch-2.9 since all the critical/blocker issues are now
>>> resolved.
>>> We plan to perform some sanity checks for the rest of the week and cut
>>> branch-2.9.0 and push out an RC0 by the end of the week.
>>>
>>> Kindly refrain from committing to branch-2.9 without giving us a heads
>>> up.
>>>
>>> @Junping, lets move the jdiff conversation to separate thread.
>>>
>>> Cheers
>>> -Arun/Subru
>>>
>>> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan 
>>> wrote:
>>>
>>> > We want to give heads up that we are going to cut branch-2.9 tomorrow
>>> > morning.
>>> >
>>> > We are down to 3 blockers and they all are close to being committed
>>> > (thanks everyone):
>>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>>> >
>>> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which
>>> are
>>> > close to completion.
>>> >
>>> > Folks who are working/reviewing these, kindly prioritize accordingly so
>>> > that we can make the release on time.
>>> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>>> >
>>> > Thanks in advance!
>>> >
>>> > -Subru/Arun
>>> >
>>> >
>>> >
>>>
>>
>>
>


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Cutting branch-2.9

[Junping Du]

Hi Subru and Arun,
 Thanks for moving forward with 2.9 release. Is the first cut of 2.9 
release supposed to be a stable version or just an alpha version? If it is 
supposed to be a stable version, we should run jdiff test and check for API 
compatibility before releasing out. Please let me know if you need any help 
here.

Thanks,

Junping

From: Subru Krishnan 
Sent: Monday, October 30, 2017 12:39 PM
To: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Cc: Arun Suresh
Subject: Cutting branch-2.9

We want to give heads up that we are going to cut branch-2.9 tomorrow
morning.

We are down to 3 blockers and they all are close to being committed (thanks
everyone):
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release

There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are
close to completion.

Folks who are working/reviewing these, kindly prioritize accordingly so
that we can make the release on time.
https://issues.apache.org/jira/browse/YARN-7398?filter=12342468

Thanks in advance!

-Subru/Arun



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[RESULT] [VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Junping Du]

??Thanks again for all who verified and voted!

I give my binding +1 to conclude the vote for 2.8.2 RC1, based on:
- Build from source and verify signatures
- Deploy pseudo-distributed cluster and run some simple job, like: pi, sleep, 
etc.
- Verify UI of daemons, like: NameNode, ResourceManager, NodeManager, etc.

Now, we have:

7 binding +1s, from:
 John Zhuge, Jason Lowe, Chris Douglas, Wangda Tan, Ravi Prakash, Eric 
Payne, Junping Du

10 non-binding +1s, from:
Hanisha Koneru, Wei Yan, Brahma Reddy Battula, Shane Kumpf, Ajay Kumar, 
Bharat Viswanadham, Mukul Kumar Singh, Eric Badger, Bibinchundatt, Rakesh 
Radhakrishnan

and no -1s.

So I am glad to announce that the vote of 2.8.2 RC1 passes.

Thanks everyone listed above who tried the release candidate and vote. Also, 
kudos to all who ever help with 2.8.2 release effort in all kinds of ways- 
especially the Yahoo! guys who deployed 2.8 in production environment and 
identify many issues with fixes. Also, Shane, Miklos and others to help with 
docker container effort during RC stage.

I'll push the release bits and send out an announcement for 2.8.2 soon.


Thanks,

Junping?


From: Eric Payne <erichadoo...@yahoo.com>
Sent: Tuesday, October 24, 2017 3:29 PM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

+1 (binding)

Thanks a lot, Junping!

I built and installed the source on a 6-node pseudo cluster. I simple sleep and 
streaming jobs that exercised intra-queue and inter-queue preemption, and used 
user weights.

-Eric


From: Junping Du <j...@hortonworks.com>
To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>
Sent: Thursday, October 19, 2017 7:43 PM
Subject: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

Hi folks,
I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.

Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and will 
be the latest stable/production release for Apache Hadoop - it includes 315 new 
fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC1<http://home.apache.org/~junping_du/hadoop-2.8.2-RC0>

  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
66c47f2a01ad9637879e95f80c41f798373828fb

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org/> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1064<https://repository.apache.org/content/repositories/orgapachehadoop-1062>

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 10/24/2017 6pm PST time.

Thanks,

Junping

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Junping Du]

In general, the "solid evidence" of memory leak comes from analysis of 
heapdump, jastack, gc log, etc. In many cases, we can locate/conclude which 
piece of code are leaking memory from the analysis.

Unfortunately, I cannot find any conclusion from previous comments and it even 
cannot tell which daemons/components of HDFS consumes unexpected high memory. 
Don't sounds like a solid bug report to me.



Thanks,?


Junping



From: Sean Busbey <bus...@cloudera.com>
Sent: Tuesday, October 24, 2017 2:20 PM
To: Junping Du
Cc: Allen Wittenauer; Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

Just curious, Junping what would "solid evidence" look like? Is the supposition 
here that the memory leak is within HDFS test code rather than library runtime 
code? How would such a distinction be shown?

On Tue, Oct 24, 2017 at 4:06 PM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Allen,
 Do we have any solid evidence to show the HDFS unit tests going through 
the roof are due to serious memory leak by HDFS? Normally, I don't expect 
memory leak are identified in our UTs - mostly, it (test jvm gone) is just 
because of test or deployment issues.
 Unless there is concrete evidence, my concern on seriously memory leak for 
HDFS on 2.8 is relatively low given some companies (Yahoo, Alibaba, etc.) have 
deployed 2.8 on large production environment for months. Non-serious memory 
leak (like forgetting to close stream in non-critical path, etc.) and other 
non-critical bugs always happens here and there that we have to live with.

Thanks,

Junping


From: Allen Wittenauer 
<a...@effectivemachines.com<mailto:a...@effectivemachines.com>>
Sent: Tuesday, October 24, 2017 8:27 AM
To: Hadoop Common
Cc: Hdfs-dev; 
mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>
Subject: Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

> On Oct 23, 2017, at 12:50 PM, Allen Wittenauer 
> <a...@effectivemachines.com<mailto:a...@effectivemachines.com>> wrote:
>
>
>
> With no other information or access to go on, my current hunch is that one of 
> the HDFS unit tests is ballooning in memory size.  The easiest way to kill a 
> Linux machine is to eat all of the RAM, thanks to overcommit and that's what 
> this "feels" like.
>
> Someone should verify if 2.8.2 has the same issues before a release goes out 
> ...


FWIW, I ran 2.8.2 last night and it has the same problems.

Also: the node didn't die!  Looking through the workspace (so the next 
run will destroy them), two sets of logs stand out:

https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/ws/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

and

https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/ws/sourcedir/hadoop-hdfs-project/hadoop-hdfs/

It looks like my hunch is correct:  RAM in the HDFS unit tests are 
going through the roof.  It's also interesting how MANY log files there are.  
Is surefire not picking up that jobs are dying?  Maybe not if memory is getting 
tight.

Anyway, at the point, branch-2.8 and higher are probably fubar'd. 
Additionally, I've filed YETUS-561 so that Yetus-controlled Docker containers 
can have their RAM limits set in order to prevent more nodes going catatonic.



-
To unsubscribe, e-mail: 
yarn-dev-unsubscr...@hadoop.apache.org<mailto:yarn-dev-unsubscr...@hadoop.apache.org>
For additional commands, e-mail: 
yarn-dev-h...@hadoop.apache.org<mailto:yarn-dev-h...@hadoop.apache.org>



-
To unsubscribe, e-mail: 
common-dev-unsubscr...@hadoop.apache.org<mailto:common-dev-unsubscr...@hadoop.apache.org>
For additional commands, e-mail: 
common-dev-h...@hadoop.apache.org<mailto:common-dev-h...@hadoop.apache.org>




--
busbey

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Junping Du]

Allen,
 Do we have any solid evidence to show the HDFS unit tests going through 
the roof are due to serious memory leak by HDFS? Normally, I don't expect 
memory leak are identified in our UTs - mostly, it (test jvm gone) is just 
because of test or deployment issues. 
 Unless there is concrete evidence, my concern on seriously memory leak for 
HDFS on 2.8 is relatively low given some companies (Yahoo, Alibaba, etc.) have 
deployed 2.8 on large production environment for months. Non-serious memory 
leak (like forgetting to close stream in non-critical path, etc.) and other 
non-critical bugs always happens here and there that we have to live with.

Thanks,

Junping


From: Allen Wittenauer 
Sent: Tuesday, October 24, 2017 8:27 AM
To: Hadoop Common
Cc: Hdfs-dev; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

> On Oct 23, 2017, at 12:50 PM, Allen Wittenauer  
> wrote:
>
>
>
> With no other information or access to go on, my current hunch is that one of 
> the HDFS unit tests is ballooning in memory size.  The easiest way to kill a 
> Linux machine is to eat all of the RAM, thanks to overcommit and that’s what 
> this “feels” like.
>
> Someone should verify if 2.8.2 has the same issues before a release goes out …


FWIW, I ran 2.8.2 last night and it has the same problems.

Also: the node didn’t die!  Looking through the workspace (so the next 
run will destroy them), two sets of logs stand out:

https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/ws/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

and

https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/ws/sourcedir/hadoop-hdfs-project/hadoop-hdfs/

It looks like my hunch is correct:  RAM in the HDFS unit tests are 
going through the roof.  It’s also interesting how MANY log files there are.  
Is surefire not picking up that jobs are dying?  Maybe not if memory is getting 
tight.

Anyway, at the point, branch-2.8 and higher are probably fubar’d. 
Additionally, I’ve filed YETUS-561 so that Yetus-controlled Docker containers 
can have their RAM limits set in order to prevent more nodes going catatonic.



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Junping Du]

Thanks Jason and all for trying the release RC and vote!

A kindly reminder: we will close the vote on tomorrow night (PST time), so 
please do the verification and vote before that time if you plan to. Thanks!?


Thanks,


Junping



From: Jason Lowe <jl...@oath.com>
Sent: Monday, October 23, 2017 2:49 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

My apologies, false alarm on the CHANGES.md and RELEASENOTES.md.  I was in the 
process of reviewing the release and was interrupted, and when I resumed I 
thought I had already downloaded the CHANGES and RELEASENOTES, but in fact they 
were the old versions from a prior review of 2.8.0.  I reviewed both of them 
for 2.8.2 (for real this time!) and they look correct.  Again my apologies for 
the confusion.

Jason

On Mon, Oct 23, 2017 at 3:26 PM, Jason Lowe 
<jl...@oath.com<mailto:jl...@oath.com>> wrote:
+1 (binding)

- Verified signatures and digests
- Performed a native build from source
- Deployed to a single-node cluster
- Ran some sample jobs

The CHANGES.md and RELEASENOTES.md both refer to release 2.8.0 instead of 
2.8.2, and I do not see the list of JIRAs in CHANGES.md that have been 
committed since 2.8.1.  Since we're voting on the source bits rather than the 
change log I kept my vote as a +1 as I do see the 2.8.2 changes in the source 
code.

Jason


On Thu, Oct 19, 2017 at 7:42 PM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi folks,
 I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.

 Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
will be the latest stable/production release for Apache Hadoop - it includes 
315 new fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical 
issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC1<http://home.apache.org/~junping_du/hadoop-2.8.2-RC0>

  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
66c47f2a01ad9637879e95f80c41f798373828fb

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org><http://repository.apache.org/>
 at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1064<https://repository.apache.org/content/repositories/orgapachehadoop-1062>

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 10/24/2017 6pm PST time.

Thanks,

Junping

[VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Junping Du]

Hi folks,
 I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.

 Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
will be the latest stable/production release for Apache Hadoop - it includes 
315 new fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical 
issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC1

  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
66c47f2a01ad9637879e95f80c41f798373828fb

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1064

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 10/24/2017 6pm PST time.

Thanks,

Junping

Re: [Update] Apache Hadoop 2.8.2 Release Status

[Junping Du]

A quick update: the last patch (YARN-7230) for docker container support in 2.8 
just get committed yesterday. Now there is no left blocker/critical issues for 
2.8.2 and I checked all landed commits are matching with JIRA's fix version. 
With kicking off a new RC build, I will publish RC bits for vote once the build 
process get finished. In the mean time, please hold on any commits to 
branch-2.8.2 unless it really belongs to a blocker and please ping me ahead. 

Thanks all for your patience!

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Friday, September 22, 2017 5:57 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: Shane Kumpf; Miklos Szegedi; Varun Vasudev
Subject: [Update] Apache Hadoop 2.8.2 Release Status

Hi folks,
 I would like to give you a quick update on 2.8.2 release status:

- First release candidate (RC0) is published over the last weekend, but several 
docker container blockers (bugs, documents, etc.)
 are reported so we decided to cancel the RC0 for vote.

- New coming release blockers (for docker container support) are YARN-7034 
(just committed), YARN-6623, YARN-6930 and YARN-7230.
Shane, Miklos and Varun are actively working on this. Appreciate the effort 
here!

- I will kick off new release candidate (RC1) once these blockers are resolved.

To all committers, branch-2.8.2 is still open for blocker/critical issues 
landing, but for major/minor/trivial issues, please commit to branch-2.8 and 
marked the fixed version as 2.8.3.

Thanks all for heads up. Have a good weekend!


Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Tuesday, September 5, 2017 2:57 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

I assume the quiet over the holiday means we agreed to move forward without 
taking HADOOP-14439 into 2.8.2.
There is a new release building (docker based) issue could be related to 
HADOOP-14474 where we removed oracle java 7 installer due to recent download 
address/contract change by Oracle. The build refuse to work - report as 
JAVA_HOME issue, but hard coded my local java home in create-release or 
Dockerfile doesn't help so we may need to add java 7 installation back (no 
matter Oracle JDK 7 or openJDK 7).
Filed HADOOP-14842 with more details to track as blocker for 2.8.2.

Thanks,

Junping
____
From: Junping Du <j...@hortonworks.com>
Sent: Friday, September 1, 2017 12:37 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

This issue (HADOOP-14439) is out of my radar given it is marked as Minor 
priority. If my understanding is correct, here is a trade-off between security 
and backward compatibility. IMO, priority of security is generally higher than 
backward compatibility especially 2.8.0 is still non-production release.
I think we should skip this for 2.8.2 in case it doesn't break compatibility 
from 2.7.x. Thoughts?

Thanks,

Junping

From: larry mccay <lmc...@apache.org>
Sent: Friday, September 1, 2017 10:55 AM
To: Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

If we do "fix" this in 2.8.2 we should seriously consider not doing so in
3.0.
This is a very poor practice.

I can see an argument for backward compatibility in 2.8.x line though.

On Fri, Sep 1, 2017 at 1:41 PM, Steve Loughran <ste...@hortonworks.com>
wrote:

> One thing we need to consider is
>
> HADOOP-14439: regression: secret stripping from S3x URIs breaks some
> downstream code
>
> Hadoop 2.8 has a best-effort attempt to strip out secrets from the
> toString() value of an s3a or s3n path where someone has embedded them in
> the URI; this has caused problems in some uses, specifically: when people
> use secrets this way (bad) and assume that you can round trip paths to
> string and back
>
> Should we fix this? If so, Hadoop 2.8.2 is the time to do it
>
>
> > On 1 Sep 2017, at 11:14, Junping Du <j...@hortonworks.com> wrote:
> >
> > HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we
> are clean on blocker/critical issues now.
> > I finish practice of going through JACC report and no more incompatible
> public API changes get found between 2.8.2 and 2.7.4. Also I check commit
> history and fixed 10+ commits which are missing from branch-2.8.2 for s

[Update] Apache Hadoop 2.8.2 Release Status

[Junping Du]

Hi folks,
 I would like to give you a quick update on 2.8.2 release status:

- First release candidate (RC0) is published over the last weekend, but several 
docker container blockers (bugs, documents, etc.)
 are reported so we decided to cancel the RC0 for vote.

- New coming release blockers (for docker container support) are YARN-7034 
(just committed), YARN-6623, YARN-6930 and YARN-7230. 
Shane, Miklos and Varun are actively working on this. Appreciate the effort 
here!

- I will kick off new release candidate (RC1) once these blockers are resolved.

To all committers, branch-2.8.2 is still open for blocker/critical issues 
landing, but for major/minor/trivial issues, please commit to branch-2.8 and 
marked the fixed version as 2.8.3.

Thanks all for heads up. Have a good weekend!


Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Tuesday, September 5, 2017 2:57 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

I assume the quiet over the holiday means we agreed to move forward without 
taking HADOOP-14439 into 2.8.2.
There is a new release building (docker based) issue could be related to 
HADOOP-14474 where we removed oracle java 7 installer due to recent download 
address/contract change by Oracle. The build refuse to work - report as 
JAVA_HOME issue, but hard coded my local java home in create-release or 
Dockerfile doesn't help so we may need to add java 7 installation back (no 
matter Oracle JDK 7 or openJDK 7).
Filed HADOOP-14842 with more details to track as blocker for 2.8.2.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Friday, September 1, 2017 12:37 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

This issue (HADOOP-14439) is out of my radar given it is marked as Minor 
priority. If my understanding is correct, here is a trade-off between security 
and backward compatibility. IMO, priority of security is generally higher than 
backward compatibility especially 2.8.0 is still non-production release.
I think we should skip this for 2.8.2 in case it doesn't break compatibility 
from 2.7.x. Thoughts?

Thanks,

Junping

From: larry mccay <lmc...@apache.org>
Sent: Friday, September 1, 2017 10:55 AM
To: Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

If we do "fix" this in 2.8.2 we should seriously consider not doing so in
3.0.
This is a very poor practice.

I can see an argument for backward compatibility in 2.8.x line though.

On Fri, Sep 1, 2017 at 1:41 PM, Steve Loughran <ste...@hortonworks.com>
wrote:

> One thing we need to consider is
>
> HADOOP-14439: regression: secret stripping from S3x URIs breaks some
> downstream code
>
> Hadoop 2.8 has a best-effort attempt to strip out secrets from the
> toString() value of an s3a or s3n path where someone has embedded them in
> the URI; this has caused problems in some uses, specifically: when people
> use secrets this way (bad) and assume that you can round trip paths to
> string and back
>
> Should we fix this? If so, Hadoop 2.8.2 is the time to do it
>
>
> > On 1 Sep 2017, at 11:14, Junping Du <j...@hortonworks.com> wrote:
> >
> > HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we
> are clean on blocker/critical issues now.
> > I finish practice of going through JACC report and no more incompatible
> public API changes get found between 2.8.2 and 2.7.4. Also I check commit
> history and fixed 10+ commits which are missing from branch-2.8.2 for some
> reason. So, the current branch-2.8.2 should be good to go for RC stage, and
> I will kick off our first RC tomorrow.
> > In the meanwhile, please don't land any commits to branch-2.8.2 since
> now. If some issues really belong to blocker, please ping me on the JIRA
> before doing any commits. branch-2.8 is still open for landing. Thanks for
> your cooperation!
> >
> >
> > Thanks,
> >
> > Junping
> >
> > 
> > From: Junping Du <j...@hortonworks.com>
> > Sent: Wednesday, August 30, 2017 12:35 AM
> > To: Brahma Reddy Battula; common-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> > Subject: Re: Apache Hadoop 2.8.2 Release Plan
> >
> > Thanks Brahma for co

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Junping Du]

Miklos just ping me offline that a security fix should get landed to resolve a 
docker runtime issue. I will retrieval RC0 for security fixes landing.
In the mean while, if people here really think a document here is necessary 
(although not enough verification to work as an alpha feature) and can work out 
a patch soon, I am open to accept it. 

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Monday, September 11, 2017 5:32 PM
To: Daniel Templeton; Chris Douglas
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

Thanks Daniel for volunteering for documentation effort.
I suspect the problem we are facing here is not a simply documentation truncate 
effort, but indeed a incomplete feature problem. The umbrella JIRA YARN-3611 
has 31 issues are marked as resolved but only 9 patches get backport to 
branch-2.8 no matter intentioned or not. So far, I haven't heard anyone claimed 
that they are deploying/testing 2.8 release without cherry pick additional 
patches for enabling new docker executor runtime.
IMO, it is better to treat this feature in 2.8 as incomplete feature instead of 
an alpha/experimental feature, especially it is out of our previous 2.8 scope. 
If so, instead of documenting something misleading, we should keep new 
settings/configurations private (as it is now) to get rid of getting enabled by 
users occasionally.
Thoughts?


Thanks,

Junping

From: Daniel Templeton <dan...@cloudera.com>
Sent: Monday, September 11, 2017 4:45 PM
To: Chris Douglas; Junping Du
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

YARN-6622 is now committed to 2.9.  We could backport YARN-5258 and
YARN-6622 for 2.8, but it'll take some editing.  We'll have to check to
see what features are unsupported in 2.8 and remove those from the
docs.  Not a huge effort overall, though.  Probably a hour's work.  I
may have time to try do it later this week.  Anyone else want to volunteer?

Daniel

On 9/11/17 3:01 PM, Chris Douglas wrote:
> On Mon, Sep 11, 2017 at 2:52 PM, Junping Du <j...@hortonworks.com> wrote:
>> I don't think this -1 is reasonable, because:
>> - If you look at YARN-6622 closely, it targets to fix a problematic 
>> documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch 
>> only. It means it targets to fix a problem that 2.8.2 never exists.
> ...we're not going to document security implications- which include
> escalations to root- because we don't have _any_ documentation? Why
> don't we backport the documentation?
>
>> - New docker container support (replace of old DockerContainerExectutor) is 
>> still an alpha feature now which doesn't highlight in 2.8 major 
>> features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
>> adding documentation here is also not a blocker.
> YARN-6622 is *documenting* the fact that this is an alpha feature and
> that it shouldn't be enabled in secure environments. How are users
> supposed to make this determination without it?
>
>> Vote still continue until a real blocker comes.
> Soright. I remain -1. -C
>
>> ____________
>> From: Chris Douglas <cdoug...@apache.org>
>> Sent: Monday, September 11, 2017 12:00 PM
>> To: Junping Du
>> Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
>> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>
>> -1 (binding)
>>
>> I don't think we should release this without YARN-6622.
>>
>> Since this doesn't happen often: a -1 in this case is NOT a veto.
>> Releases are approved by majority vote of the PMC. -C
>>
>> On Mon, Sep 11, 2017 at 11:45 AM, Junping Du <j...@hortonworks.com> wrote:
>>> Thanks Mikols for notifying on this. I think docker support is general 
>>> known as alpha feature so document it as experimental is nice to have but 
>>> not a blocker for 2.8.2. I also noticed that our 2.7.x document 
>>> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>>>  without mentioning docker support is experimental. We may need to fix that 
>>> as well in following releases.
>>>
>>> I can also add it (mentioning docker container support feature is 
>>> experimental) to release message in public website just like previous 
>>> release we call 2.7.0/2.8.0 as non-production release.
>&

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Junping Du]

Thanks Daniel for volunteering for documentation effort. 
I suspect the problem we are facing here is not a simply documentation truncate 
effort, but indeed a incomplete feature problem. The umbrella JIRA YARN-3611 
has 31 issues are marked as resolved but only 9 patches get backport to 
branch-2.8 no matter intentioned or not. So far, I haven't heard anyone claimed 
that they are deploying/testing 2.8 release without cherry pick additional 
patches for enabling new docker executor runtime.
IMO, it is better to treat this feature in 2.8 as incomplete feature instead of 
an alpha/experimental feature, especially it is out of our previous 2.8 scope. 
If so, instead of documenting something misleading, we should keep new 
settings/configurations private (as it is now) to get rid of getting enabled by 
users occasionally.
Thoughts?


Thanks,

Junping

From: Daniel Templeton <dan...@cloudera.com>
Sent: Monday, September 11, 2017 4:45 PM
To: Chris Douglas; Junping Du
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

YARN-6622 is now committed to 2.9.  We could backport YARN-5258 and
YARN-6622 for 2.8, but it'll take some editing.  We'll have to check to
see what features are unsupported in 2.8 and remove those from the
docs.  Not a huge effort overall, though.  Probably a hour's work.  I
may have time to try do it later this week.  Anyone else want to volunteer?

Daniel

On 9/11/17 3:01 PM, Chris Douglas wrote:
> On Mon, Sep 11, 2017 at 2:52 PM, Junping Du <j...@hortonworks.com> wrote:
>> I don't think this -1 is reasonable, because:
>> - If you look at YARN-6622 closely, it targets to fix a problematic 
>> documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch 
>> only. It means it targets to fix a problem that 2.8.2 never exists.
> ...we're not going to document security implications- which include
> escalations to root- because we don't have _any_ documentation? Why
> don't we backport the documentation?
>
>> - New docker container support (replace of old DockerContainerExectutor) is 
>> still an alpha feature now which doesn't highlight in 2.8 major 
>> features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
>> adding documentation here is also not a blocker.
> YARN-6622 is *documenting* the fact that this is an alpha feature and
> that it shouldn't be enabled in secure environments. How are users
> supposed to make this determination without it?
>
>> Vote still continue until a real blocker comes.
> Soright. I remain -1. -C
>
>> ____
>> From: Chris Douglas <cdoug...@apache.org>
>> Sent: Monday, September 11, 2017 12:00 PM
>> To: Junping Du
>> Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
>> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>
>> -1 (binding)
>>
>> I don't think we should release this without YARN-6622.
>>
>> Since this doesn't happen often: a -1 in this case is NOT a veto.
>> Releases are approved by majority vote of the PMC. -C
>>
>> On Mon, Sep 11, 2017 at 11:45 AM, Junping Du <j...@hortonworks.com> wrote:
>>> Thanks Mikols for notifying on this. I think docker support is general 
>>> known as alpha feature so document it as experimental is nice to have but 
>>> not a blocker for 2.8.2. I also noticed that our 2.7.x document 
>>> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>>>  without mentioning docker support is experimental. We may need to fix that 
>>> as well in following releases.
>>>
>>> I can also add it (mentioning docker container support feature is 
>>> experimental) to release message in public website just like previous 
>>> release we call 2.7.0/2.8.0 as non-production release.
>>>
>>> I think vote should continue until we could find a real blocker.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Junping
>>>
>>>
>>> 
>>> From: Miklos Szegedi <miklos.szeg...@cloudera.com>
>>> Sent: Monday, September 11, 2017 10:07 AM
>>> To: Mingliang Liu
>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org; 
>>> yarn-...@hadoop.apache.org; junping_du; Junping Du
>>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>>
>>> Hello Junping,
>>>
>>> Thank you for working on this. Shoul

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Junping Du]

I don't think this -1 is reasonable, because:
- If you look at YARN-6622 closely, it targets to fix a problematic 
documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch only. 
It means it targets to fix a problem that 2.8.2 never exists.
- New docker container support (replace of old DockerContainerExectutor) is 
still an alpha feature now which doesn't highlight in 2.8 major 
features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
adding documentation here is also not a blocker.

Vote still continue until a real blocker comes.

Thanks,

Junping


From: Chris Douglas <cdoug...@apache.org>
Sent: Monday, September 11, 2017 12:00 PM
To: Junping Du
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

-1 (binding)

I don't think we should release this without YARN-6622.

Since this doesn't happen often: a -1 in this case is NOT a veto.
Releases are approved by majority vote of the PMC. -C

On Mon, Sep 11, 2017 at 11:45 AM, Junping Du <j...@hortonworks.com> wrote:
> Thanks Mikols for notifying on this. I think docker support is general known 
> as alpha feature so document it as experimental is nice to have but not a 
> blocker for 2.8.2. I also noticed that our 2.7.x document 
> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>  without mentioning docker support is experimental. We may need to fix that 
> as well in following releases.
>
> I can also add it (mentioning docker container support feature is 
> experimental) to release message in public website just like previous release 
> we call 2.7.0/2.8.0 as non-production release.
>
> I think vote should continue until we could find a real blocker.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Miklos Szegedi <miklos.szeg...@cloudera.com>
> Sent: Monday, September 11, 2017 10:07 AM
> To: Mingliang Liu
> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; junping_du; Junping Du
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>
> Hello Junping,
>
> Thank you for working on this. Should not YARN-6622 be addressed first? 
> "Summary: Document Docker work as experimental".
>
> Thank you,
> Miklos
>
>
> On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
> <lium...@gmail.com<mailto:lium...@gmail.com>> wrote:
> Thanks Junping for doing this!
>
> +1 (non-binding)
>
> - Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
> - Build package using maven (skipping tests) with Java 8
> - Spin up a test cluster in Docker containers having 1 master node (NN/RM) 
> and 3 slave nodes (DN/NM)
> - Operate the basic HDFS/YARN operations from command line, both client and 
> admin
> - Check NN/RM Web UI
> - Run distcp to copy files from/to local and HDFS
> - Run hadoop mapreduce examples: grep and wordcount
> - Check the HDFS service logs
>
> All looked good to me.
>
> Mingliang
>
>> On Sep 10, 2017, at 5:00 PM, Junping Du 
>> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
>>
>> Hi folks,
>> With fix of HADOOP-14842 get in, I've created our first release 
>> candidate (RC0) for Apache Hadoop 2.8.2.
>>
>> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
>> will be the latest stable/production release for Apache Hadoop - it includes 
>> 305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
>> issues.
>>
>>  More information about the 2.8.2 release plan can be found here: 
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>
>>  New RC is available at: 
>> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>>
>>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
>> e6597fe3000b06847d2bf55f2bab81770f4b2505
>>
>>  The maven artifacts are available via 
>> repository.apache.org<http://repository.apache.org> at: 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>>
>>  Please try the release and vote; the vote will run for the usual 5 
>> days, ending on 09/15/2017 5pm PST time.
>>
>> Thanks,
>>
>> Junping
>>
>
>
> -
> To unsubscribe, e-mail: 
> mapreduce-dev-unsubscr...@hadoop.apache.org<mailto:mapreduce-dev-unsubscr...@hadoop.apache.org>
> For additional commands, e-mail: 
>

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Junping Du]

Thanks Mikols for notifying on this. I think docker support is general known as 
alpha feature so document it as experimental is nice to have but not a blocker 
for 2.8.2. I also noticed that our 2.7.x document 
(https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
 without mentioning docker support is experimental. We may need to fix that as 
well in following releases.

I can also add it (mentioning docker container support feature is experimental) 
to release message in public website just like previous release we call 
2.7.0/2.8.0 as non-production release.

I think vote should continue until we could find a real blocker.


Thanks,


Junping



From: Miklos Szegedi <miklos.szeg...@cloudera.com>
Sent: Monday, September 11, 2017 10:07 AM
To: Mingliang Liu
Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; junping_du; Junping Du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

Hello Junping,

Thank you for working on this. Should not YARN-6622 be addressed first? 
"Summary: Document Docker work as experimental".

Thank you,
Miklos


On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
<lium...@gmail.com<mailto:lium...@gmail.com>> wrote:
Thanks Junping for doing this!

+1 (non-binding)

- Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
- Build package using maven (skipping tests) with Java 8
- Spin up a test cluster in Docker containers having 1 master node (NN/RM) and 
3 slave nodes (DN/NM)
- Operate the basic HDFS/YARN operations from command line, both client and 
admin
- Check NN/RM Web UI
- Run distcp to copy files from/to local and HDFS
- Run hadoop mapreduce examples: grep and wordcount
- Check the HDFS service logs

All looked good to me.

Mingliang

> On Sep 10, 2017, at 5:00 PM, Junping Du 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
>
> Hi folks,
> With fix of HADOOP-14842 get in, I've created our first release candidate 
> (RC0) for Apache Hadoop 2.8.2.
>
> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
> will be the latest stable/production release for Apache Hadoop - it includes 
> 305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
> issues.
>
>  More information about the 2.8.2 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>  New RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>
>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
> e6597fe3000b06847d2bf55f2bab81770f4b2505
>
>  The maven artifacts are available via 
> repository.apache.org<http://repository.apache.org> at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>
>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 09/15/2017 5pm PST time.
>
> Thanks,
>
> Junping
>


-
To unsubscribe, e-mail: 
mapreduce-dev-unsubscr...@hadoop.apache.org<mailto:mapreduce-dev-unsubscr...@hadoop.apache.org>
For additional commands, e-mail: 
mapreduce-dev-h...@hadoop.apache.org<mailto:mapreduce-dev-h...@hadoop.apache.org>

[VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Junping Du]

Hi folks,
 With fix of HADOOP-14842 get in, I've created our first release candidate 
(RC0) for Apache Hadoop 2.8.2.

 Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
will be the latest stable/production release for Apache Hadoop - it includes 
305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC0

  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
e6597fe3000b06847d2bf55f2bab81770f4b2505

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1062

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 09/15/2017 5pm PST time.

Thanks,

Junping

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

I assume the quiet over the holiday means we agreed to move forward without 
taking HADOOP-14439 into 2.8.2.
There is a new release building (docker based) issue could be related to 
HADOOP-14474 where we removed oracle java 7 installer due to recent download 
address/contract change by Oracle. The build refuse to work - report as 
JAVA_HOME issue, but hard coded my local java home in create-release or 
Dockerfile doesn't help so we may need to add java 7 installation back (no 
matter Oracle JDK 7 or openJDK 7). 
Filed HADOOP-14842 with more details to track as blocker for 2.8.2.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Friday, September 1, 2017 12:37 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

This issue (HADOOP-14439) is out of my radar given it is marked as Minor 
priority. If my understanding is correct, here is a trade-off between security 
and backward compatibility. IMO, priority of security is generally higher than 
backward compatibility especially 2.8.0 is still non-production release.
I think we should skip this for 2.8.2 in case it doesn't break compatibility 
from 2.7.x. Thoughts?

Thanks,

Junping

From: larry mccay <lmc...@apache.org>
Sent: Friday, September 1, 2017 10:55 AM
To: Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

If we do "fix" this in 2.8.2 we should seriously consider not doing so in
3.0.
This is a very poor practice.

I can see an argument for backward compatibility in 2.8.x line though.

On Fri, Sep 1, 2017 at 1:41 PM, Steve Loughran <ste...@hortonworks.com>
wrote:

> One thing we need to consider is
>
> HADOOP-14439: regression: secret stripping from S3x URIs breaks some
> downstream code
>
> Hadoop 2.8 has a best-effort attempt to strip out secrets from the
> toString() value of an s3a or s3n path where someone has embedded them in
> the URI; this has caused problems in some uses, specifically: when people
> use secrets this way (bad) and assume that you can round trip paths to
> string and back
>
> Should we fix this? If so, Hadoop 2.8.2 is the time to do it
>
>
> > On 1 Sep 2017, at 11:14, Junping Du <j...@hortonworks.com> wrote:
> >
> > HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we
> are clean on blocker/critical issues now.
> > I finish practice of going through JACC report and no more incompatible
> public API changes get found between 2.8.2 and 2.7.4. Also I check commit
> history and fixed 10+ commits which are missing from branch-2.8.2 for some
> reason. So, the current branch-2.8.2 should be good to go for RC stage, and
> I will kick off our first RC tomorrow.
> > In the meanwhile, please don't land any commits to branch-2.8.2 since
> now. If some issues really belong to blocker, please ping me on the JIRA
> before doing any commits. branch-2.8 is still open for landing. Thanks for
> your cooperation!
> >
> >
> > Thanks,
> >
> > Junping
> >
> > 
> > From: Junping Du <j...@hortonworks.com>
> > Sent: Wednesday, August 30, 2017 12:35 AM
> > To: Brahma Reddy Battula; common-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> > Subject: Re: Apache Hadoop 2.8.2 Release Plan
> >
> > Thanks Brahma for comment on this thread. To be clear, I always update
> branch version just before RC kicking off.
> >
> > For 2.8.2 release, I don't have plan to involve big top or other
> third-party test tools. As always, we will rely on test/verify efforts from
> community especially from large deployed production cluster - as far as I
> know,  there are already several companies. like: Yahoo!, Alibaba, etc.
> already deploy 2.8 release in large production clusters for months which
> give me more confidence on 2.8.2.
> >
> >
> > Here is more update on 2.8.2 release:
> >
> > Blocker issues:
> >
> >-  A new blocker YARN-7076 get reported and fixed by Jian He through
> last weekend.
> >
> >-  Another new blocker - HADOOP-14814 get identified from my latest
> jdiff run against 2.7.4. The simple fix on an incompatible API change
> should get commit soon.
> >
> >
> > Critical issues:
> >
> >-  YARN-7083 already get committed. Thanks Jason for reporting the
> issue and delivering the fix.
> >
> >-  Y

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

This issue (HADOOP-14439) is out of my radar given it is marked as Minor 
priority. If my understanding is correct, here is a trade-off between security 
and backward compatibility. IMO, priority of security is generally higher than 
backward compatibility especially 2.8.0 is still non-production release. 
I think we should skip this for 2.8.2 in case it doesn't break compatibility 
from 2.7.x. Thoughts?

Thanks,

Junping

From: larry mccay <lmc...@apache.org>
Sent: Friday, September 1, 2017 10:55 AM
To: Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

If we do "fix" this in 2.8.2 we should seriously consider not doing so in
3.0.
This is a very poor practice.

I can see an argument for backward compatibility in 2.8.x line though.

On Fri, Sep 1, 2017 at 1:41 PM, Steve Loughran <ste...@hortonworks.com>
wrote:

> One thing we need to consider is
>
> HADOOP-14439: regression: secret stripping from S3x URIs breaks some
> downstream code
>
> Hadoop 2.8 has a best-effort attempt to strip out secrets from the
> toString() value of an s3a or s3n path where someone has embedded them in
> the URI; this has caused problems in some uses, specifically: when people
> use secrets this way (bad) and assume that you can round trip paths to
> string and back
>
> Should we fix this? If so, Hadoop 2.8.2 is the time to do it
>
>
> > On 1 Sep 2017, at 11:14, Junping Du <j...@hortonworks.com> wrote:
> >
> > HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we
> are clean on blocker/critical issues now.
> > I finish practice of going through JACC report and no more incompatible
> public API changes get found between 2.8.2 and 2.7.4. Also I check commit
> history and fixed 10+ commits which are missing from branch-2.8.2 for some
> reason. So, the current branch-2.8.2 should be good to go for RC stage, and
> I will kick off our first RC tomorrow.
> > In the meanwhile, please don't land any commits to branch-2.8.2 since
> now. If some issues really belong to blocker, please ping me on the JIRA
> before doing any commits. branch-2.8 is still open for landing. Thanks for
> your cooperation!
> >
> >
> > Thanks,
> >
> > Junping
> >
> > 
> > From: Junping Du <j...@hortonworks.com>
> > Sent: Wednesday, August 30, 2017 12:35 AM
> > To: Brahma Reddy Battula; common-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> > Subject: Re: Apache Hadoop 2.8.2 Release Plan
> >
> > Thanks Brahma for comment on this thread. To be clear, I always update
> branch version just before RC kicking off.
> >
> > For 2.8.2 release, I don't have plan to involve big top or other
> third-party test tools. As always, we will rely on test/verify efforts from
> community especially from large deployed production cluster - as far as I
> know,  there are already several companies. like: Yahoo!, Alibaba, etc.
> already deploy 2.8 release in large production clusters for months which
> give me more confidence on 2.8.2.
> >
> >
> > Here is more update on 2.8.2 release:
> >
> > Blocker issues:
> >
> >-  A new blocker YARN-7076 get reported and fixed by Jian He through
> last weekend.
> >
> >-  Another new blocker - HADOOP-14814 get identified from my latest
> jdiff run against 2.7.4. The simple fix on an incompatible API change
> should get commit soon.
> >
> >
> > Critical issues:
> >
> >-  YARN-7083 already get committed. Thanks Jason for reporting the
> issue and delivering the fix.
> >
> >-  YARN-6091 get push out from 2.8.2 as issue is not a regression and
> pending for a while.
> >
> >-  Daryn is actively working on HADOOP-9747 for a while, and the
> patch are getting close to be committed. However, according to Daryn, the
> patch seems to cause some regression in some corner cases in secured
> environment (Storm auto tgt, etc.). May need some additional watch/review
> on this JIRA's fixes.
> >
> >
> >
> > My monitoring JACC report between 2.8.2 and 2.7.4 will get finish
> tomorrow. If everything is going smoothly, I am planning to kick off RC0
> around holiday (this weekend).
> >
> >
> >
> > Thanks,
> >
> >
> >
> > ​Junping
> >
> >
> >
> > 
> > From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
> > Sent: Tuesday, A

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we are 
clean on blocker/critical issues now.
I finish practice of going through JACC report and no more incompatible public 
API changes get found between 2.8.2 and 2.7.4. Also I check commit history and 
fixed 10+ commits which are missing from branch-2.8.2 for some reason. So, the 
current branch-2.8.2 should be good to go for RC stage, and I will kick off our 
first RC tomorrow.
In the meanwhile, please don't land any commits to branch-2.8.2 since now. If 
some issues really belong to blocker, please ping me on the JIRA before doing 
any commits. branch-2.8 is still open for landing. Thanks for your cooperation!


Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, August 30, 2017 12:35 AM
To: Brahma Reddy Battula; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Thanks Brahma for comment on this thread. To be clear, I always update branch 
version just before RC kicking off.

For 2.8.2 release, I don't have plan to involve big top or other third-party 
test tools. As always, we will rely on test/verify efforts from community 
especially from large deployed production cluster - as far as I know,  there 
are already several companies. like: Yahoo!, Alibaba, etc. already deploy 2.8 
release in large production clusters for months which give me more confidence 
on 2.8.2.


Here is more update on 2.8.2 release:

Blocker issues:

-  A new blocker YARN-7076 get reported and fixed by Jian He through last 
weekend.

-  Another new blocker - HADOOP-14814 get identified from my latest jdiff 
run against 2.7.4. The simple fix on an incompatible API change should get 
commit soon.


Critical issues:

-  YARN-7083 already get committed. Thanks Jason for reporting the issue 
and delivering the fix.

-  YARN-6091 get push out from 2.8.2 as issue is not a regression and 
pending for a while.

-  Daryn is actively working on HADOOP-9747 for a while, and the patch are 
getting close to be committed. However, according to Daryn, the patch seems to 
cause some regression in some corner cases in secured environment (Storm auto 
tgt, etc.). May need some additional watch/review on this JIRA's fixes.



My monitoring JACC report between 2.8.2 and 2.7.4 will get finish tomorrow. If 
everything is going smoothly, I am planning to kick off RC0 around holiday 
(this weekend).



Thanks,



​Junping




From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Tuesday, August 29, 2017 8:42 AM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Hi All

Update on 2.8.2 release status
we are down to 3 critical issues ( YARN-6091,YARN-7083,HADOOP-9747),all are 
patch available and closer to commit.
Junping is closing tracking this.

Todo:

1) Update pom.xml ..?  currently it's with 2.8.3
https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21
<https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21>2) Wiki is 
outdated, need to update the wiki..?
3) As this is going to stable release,are we planing enable Big top for 2.8.2 
testing Or Dynamometer testing (anybody from linked-in can help)..?

@Junping Du<mailto:j...@hortonworks.com>,Please correct me,if I am wrong.


--Brahma Reddy Battula
____________
From: Junping Du <j...@hortonworks.com>
Sent: Monday, August 7, 2017 2:44 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x
Issue Navigator - ASF JIRA<https://s.apache.org/JM5x>
s.apache.org
Linked Applications. Loading… Dashboards




I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping
____________
From: Junping Du <j...@hortonworks.com>
Sent: Monday, July 24, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and lo

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Thanks Brahma for comment on this thread. To be clear, I always update branch 
version just before RC kicking off.

For 2.8.2 release, I don't have plan to involve big top or other third-party 
test tools. As always, we will rely on test/verify efforts from community 
especially from large deployed production cluster - as far as I know,  there 
are already several companies. like: Yahoo!, Alibaba, etc. already deploy 2.8 
release in large production clusters for months which give me more confidence 
on 2.8.2.


Here is more update on 2.8.2 release:

Blocker issues:

-  A new blocker YARN-7076 get reported and fixed by Jian He through last 
weekend.

-  Another new blocker - HADOOP-14814 get identified from my latest jdiff 
run against 2.7.4. The simple fix on an incompatible API change should get 
commit soon.


Critical issues:

-  YARN-7083 already get committed. Thanks Jason for reporting the issue 
and delivering the fix.

-  YARN-6091 get push out from 2.8.2 as issue is not a regression and 
pending for a while.

-  Daryn is actively working on HADOOP-9747 for a while, and the patch are 
getting close to be committed. However, according to Daryn, the patch seems to 
cause some regression in some corner cases in secured environment (Storm auto 
tgt, etc.). May need some additional watch/review on this JIRA's fixes.



My monitoring JACC report between 2.8.2 and 2.7.4 will get finish tomorrow. If 
everything is going smoothly, I am planning to kick off RC0 around holiday 
(this weekend).



Thanks,



​Junping




From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Tuesday, August 29, 2017 8:42 AM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Hi All

Update on 2.8.2 release status
we are down to 3 critical issues ( YARN-6091,YARN-7083,HADOOP-9747),all are 
patch available and closer to commit.
Junping is closing tracking this.

Todo:

1) Update pom.xml ..?  currently it's with 2.8.3
https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21
<https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21>2) Wiki is 
outdated, need to update the wiki..?
3) As this is going to stable release,are we planing enable Big top for 2.8.2 
testing Or Dynamometer testing (anybody from linked-in can help)..?

@Junping Du<mailto:j...@hortonworks.com>,Please correct me,if I am wrong.


--Brahma Reddy Battula
________
From: Junping Du <j...@hortonworks.com>
Sent: Monday, August 7, 2017 2:44 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x
Issue Navigator - ASF JIRA<https://s.apache.org/JM5x>
s.apache.org
Linked Applications. Loading… Dashboards




I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping
________
From: Junping Du <j...@hortonworks.com>
Sent: Monday, July 24, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping


____________
From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behav

[jira] [Created] (MAPREDUCE-6941) The default setting doesn't work for MapReduce job

[Junping Du (JIRA)]

Junping Du created MAPREDUCE-6941:
-

 Summary: The default setting doesn't work for MapReduce job
 Key: MAPREDUCE-6941
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6941
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Junping Du
Priority: Blocker


On the deployment of hadoop 3 cluster (based on current trunk branch) with 
default settings, the MR job will get failed as following exceptions:
{noformat}
2017-08-16 13:00:03,846 INFO mapreduce.Job: Job job_1502913552390_0001 running 
in uber mode : false
2017-08-16 13:00:03,847 INFO mapreduce.Job:  map 0% reduce 0%
2017-08-16 13:00:03,864 INFO mapreduce.Job: Job job_1502913552390_0001 failed 
with state FAILED due to: Application application_1502913552390_0001 failed 2 
times due to AM Container for appattempt_1502913552390_0001_02 exited with  
exitCode: 1
Failing this attempt.Diagnostics: [2017-08-16 13:00:02.963]Exception from 
container-launch.
Container id: container_1502913552390_0001_02_01
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:994)
at org.apache.hadoop.util.Shell.run(Shell.java:887)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212)
at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:295)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:455)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:275)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:90)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

{noformat}

This is because mapreduce related jar are not added into yarn setup by default. 
To make MR job run successful, we need to add following configurations to 
yarn-site.xml now:
{noformat}

  yarn.application.classpath
  
...
/share/hadoop/mapreduce/*,
/share/hadoop/mapreduce/lib/*
...
  
{noformat}
But this config is not necessary for previous version of Hadoop. We should fix 
this issue before beta release otherwise it will be a regression for 
configuration changes.

This could be more like a YARN issue (if so, we should move), depends on how we 
fix it finally.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x

I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Monday, July 24, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping



From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If no other concerns, I will do the branch update in next 30 minutes.


Thanks,


Junping



From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Sunday, July 23, 2017 1:50 AM
To: Junping Du; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan



Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are 
missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two 
commits.Hope we'll not miss any commits which present in only in branch-2.8.2.

________
From: Junping Du <j...@hortonworks.com>
Sent: Saturday, July 22, 2017 5:57 AM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping

____________
From: Junping Du <j...@hortonworks.com>
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
>
> Thanks for suggestions, Jason and K

[jira] [Created] (MAPREDUCE-6925) CLONE - Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild

[Junping Du (JIRA)]

Junping Du created MAPREDUCE-6925:
-

 Summary: CLONE - Make Counter limits consistent across JobClient, 
MRAppMaster, and YarnChild
 Key: MAPREDUCE-6925
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6925
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
 Fix For: 2.9.0, 3.0.0-alpha1


Currently, counter limits "mapreduce.job.counters.*" handled by 
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized asymmetrically: 
on the client side, and on the AM, job.xml is ignored whereas it's taken into 
account in YarnChild.

It would be good to make the Limits job-configurable, such that max 
counters/groups is only increased when needed. With the current Limits 
implementation relying on static constants, it's going to be challenging for 
tools that submit jobs concurrently  without resorting to class loading 
isolation.

The patch that I am uploading is not perfect but demonstrates the issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Reopened] (MAPREDUCE-6199) AbstractCounters are not reset completely on deserialization

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du reopened MAPREDUCE-6199:
---

> AbstractCounters are not reset completely on deserialization
> 
>
> Key: MAPREDUCE-6199
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6199
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: mr-6199.001.patch, mr-6199.001.patch, mr-6199.002.patch
>
>
> AbstractCounters are partially reset on deserialization. This patch 
> completely resets them. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6199) AbstractCounters are not reset completely on deserialization

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du resolved MAPREDUCE-6199.
---
Resolution: Won't Fix

> AbstractCounters are not reset completely on deserialization
> 
>
> Key: MAPREDUCE-6199
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6199
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: mr-6199.001.patch, mr-6199.001.patch, mr-6199.002.patch
>
>
> AbstractCounters are partially reset on deserialization. This patch 
> completely resets them. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Reopened] (MAPREDUCE-6286) A typo in HistoryViewer makes some code useless, which causes counter limits are not reset correctly.

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du reopened MAPREDUCE-6286:
---

> A typo in HistoryViewer makes some code useless, which causes counter limits 
> are not reset correctly.
> -
>
> Key: MAPREDUCE-6286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6286
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.6.0
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6286.000.patch
>
>
> A typo in HistoryViewer makes some code useless and it causes counter limits 
> are not reset correctly.
> The typo is
> Limits.reset(conf);
> We should use jobConf instead of conf.
> With the typo, the following code becomes useless:
> {code}
>   final Path jobConfPath = new Path(jobFile.getParent(),  jobDetails[0]
>   + "_" + jobDetails[1] + "_" + jobDetails[2] + "_conf.xml");
>   final Configuration jobConf = new Configuration(conf);
> jobConf.addResource(fs.open(jobConfPath), jobConfPath.toString());
> {code}
> The code wants to load the configuration from the Job configuration file and 
> reset the Limits based on the new configuration loaded from the Job 
> configuration file. But with the typo, the Limits are reset with the old 
> configuration.
> So this typo is apparent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6288) mapred job -status fails with AccessControlException

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du resolved MAPREDUCE-6288.
---
Resolution: Done

I have revert MAPREDUCE-6286, MAPREDUCE-6199 and MAPREDUCE-5875 from trunk and 
branch-2. Resolve this jira as Done as we will reopen MAPREDUCE-5875 for a 
better solution.

> mapred job -status fails with AccessControlException 
> -
>
> Key: MAPREDUCE-6288
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-6288.002.patch, MAPREDUCE-6288-gera-001.patch, 
> MAPREDUCE-6288.patch
>
>
> After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred 
> job -status job_1427080398288_0001}}
> {noformat}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException: 
> Permission denied: user=jenkins, access=EXECUTE, 
> inode="/user/history/done":mapred:hadoop:drwxrwx---
>   at 
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
>   at 
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
>   at 
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
>   at 
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
>   at 
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>   at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>   at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
>   at 
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
>   at 
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
>   at 
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
>   at 
> or

Re:

[Junping Du]

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping



From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If no other concerns, I will do the branch update in next 30 minutes.


Thanks,


Junping



From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Sunday, July 23, 2017 1:50 AM
To: Junping Du; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan



Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are 
missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two 
commits.Hope we'll not miss any commits which present in only in branch-2.8.2.

________
From: Junping Du <j...@hortonworks.com>
Sent: Saturday, July 22, 2017 5:57 AM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping

________
From: Junping Du <j...@hortonworks.com>
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If no other concerns, I will do the branch update in next 30 minutes.


Thanks,


Junping



From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Sunday, July 23, 2017 1:50 AM
To: Junping Du; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan



Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are 
missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two 
commits.Hope we'll not miss any commits which present in only in branch-2.8.2.

________
From: Junping Du <j...@hortonworks.com>
Sent: Saturday, July 22, 2017 5:57 AM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping

________
From: Junping Du <j...@hortonworks.com>
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
>
> Jason
>
>
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
> <kih...@yahoo-inc.com.INVALID> wrote:
>
>
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
>
> On Thursday, July 20, 201

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
>
> Jason
>
>
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
> <kih...@yahoo-inc.com.INVALID> wrote:
>
>
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
>
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du <j...@hortonworks.com> 
> wrote:
>
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
>
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
>
>  Please share thoughts if you have any. Thanks!
>
> Thanks,
>
> Junping
>
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
> 
> From: Vinod Kumar Vavilapalli <vino...@apache.org>
> Sent: Thursday, July 20, 2017 1:05 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released
>
> Hi all,
>
> The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
> page: http://hadoop.apache.org/releases.html#Download
> This is a sec

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
>
> Jason
>
>
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
> <kih...@yahoo-inc.com.INVALID> wrote:
>
>
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
>
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du <j...@hortonworks.com> 
> wrote:
>
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
>
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
>
>  Please share thoughts if you have any. Thanks!
>
> Thanks,
>
> Junping
>
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
> 
> From: Vinod Kumar Vavilapalli <vino...@apache.org>
> Sent: Thursday, July 20, 2017 1:05 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released
>
> Hi all,
>
> The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
> page: http://hadoop.apache.org/releases.html#Download
> This is a security release in the 2.8.0 release line. It consists of 2.8.0 
> plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.
>
> Please note that 2.8.x release line continues to be not yet ready for 
> production use. Critical issues are being ironed out via testing and 
> downstream adoption. Production users should wait for a subsequent release in 
> the 2.8.x line.
>
> Thanks
> +Vinod
>
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Thanks for suggestions, Jason and Kihwal!
+1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
cannot be abandoned/replaced (suspect all branches are read-only now), I will 
manually merge all commits that not landed on 2.8.2 yet.

Thanks,

Junping

From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
Sent: Friday, July 21, 2017 8:17 AM
To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

+1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that are 
in branch-2.8.  There also are a lot of JIRAs that claim they are fixed in 
2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on recent 
activity in branch-2.8 would solve both of these issues, and we'd only need to 
move the handful of JIRAs that have marked themselves correctly as fixed in 
2.8.3 to be fixed in 2.8.2.

Jason


On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
<kih...@yahoo-inc.com.INVALID> wrote:


 Thanks for driving the next 2.8 release, Junping. While I was committing a 
blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
2.8.2.
Thanks,Kihwal

On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du <j...@hortonworks.com> 
wrote:

Hi all,
Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
released today which is a special security release. Now, we should work towards 
2.8.2 release which aim for production deployment. The focus obviously is to 
fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We 
currently have 13 blocker/critical issues, and 10 of them are Patch Available.

  I plan to cut an RC in a month - target for releasing before end of Aug., to 
give enough time for outstanding blocker / critical issues. Will start moving 
out any tickets that are not blockers and/or won't fit the timeline. For 
progress of releasing effort, please refer our release wiki [2].

  Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
[2] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Thursday, July 20, 2017 1:05 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released

Hi all,

The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
page: http://hadoop.apache.org/releases.html#Download
This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus 
security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.

Please note that 2.8.x release line continues to be not yet ready for 
production use. Critical issues are being ironed out via testing and downstream 
adoption. Production users should wait for a subsequent release in the 2.8.x 
line.

Thanks
+Vinod


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Hi all,
 Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
released today which is a special security release. Now, we should work towards 
2.8.2 release which aim for production deployment. The focus obviously is to 
fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We 
currently have 13 blocker/critical issues, and 10 of them are Patch Available.

   I plan to cut an RC in a month - target for releasing before end of Aug., to 
give enough time for outstanding blocker / critical issues. Will start moving 
out any tickets that are not blockers and/or won't fit the timeline. For 
progress of releasing effort, please refer our release wiki [2].

   Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
[2] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


From: Vinod Kumar Vavilapalli 
Sent: Thursday, July 20, 2017 1:05 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released

Hi all,

The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
page: http://hadoop.apache.org/releases.html#Download
This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus 
security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.

Please note that 2.8.x release line continues to be not yet ready for 
production use. Critical issues are being ironed out via testing and downstream 
adoption. Production users should wait for a subsequent release in the 2.8.x 
line.

Thanks
+Vinod


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6915) On branch-2 ResourceManager failed to start

[Junping Du (JIRA)]

Junping Du created MAPREDUCE-6915:
-

 Summary: On branch-2 ResourceManager failed to start
 Key: MAPREDUCE-6915
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6915
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.9
Reporter: Junping Du


On build against branch-2, ResourceManager get failed to start because of 
following failures:
{noformat}
2017-07-16 23:33:15,688 FATAL 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting 
ResourceManager
java.lang.NoSuchMethodError: 
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer.setMonitorInterval(I)V
at 
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer.serviceInit(ContainerAllocationExpirer.java:44)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:684)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1005)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:285)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1283)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Reopened] (MAPREDUCE-5621) mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du reopened MAPREDUCE-5621:
---

Reopen this ticket as the issue still exists on branch-2.

> mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time
> 
>
> Key: MAPREDUCE-5621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5621
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Minor
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5621.patch
>
>
> mr-jobhistory-daemon.sh executes mkdir and chown command to output the log 
> files.
> This is always executed with or without a directory. In addition, this is 
> executed not only starting daemon but also stopping daemon.
> It add "if" like hadoop-daemon.sh and yarn-daemon.sh and should control it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6897) Add Unit Test to make sure Job end notification get sent even appMaster stop get YarnRuntimeException

[Junping Du (JIRA)]

Junping Du created MAPREDUCE-6897:
-

 Summary: Add Unit Test to make sure Job end notification get sent 
even appMaster stop get YarnRuntimeException
 Key: MAPREDUCE-6897
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6897
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Junping Du
Priority: Minor


In MAPREDUCE-6895, we fix the issue that Job end notification not send due to 
YarnRuntimeException throw in appMaster stop. We need to add unit test to make 
sure we won't run into the same issue again in future.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.1 Release Plan

[Junping Du]

I just cut off branch-2.8.1 from branch-2.8 for 2.8.1 release. So from now, all 
patch target for 2.8.1 need to be checked into branch-2.8.1 in addition to 
branch-2.8.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Monday, April 03, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Apache Hadoop 2.8.1 Release Plan

Hi all,
   We just released Apache Hadoop 2.8.0 recently [1] but it is not for 
production yet due to some issues identified. Now, we should work towards 2.8.1 
release which aim for production deployment. The focus obviously is to fix 
blocker/critical issues [2], bug-fixes and *no* features / improvements.

   I plan to cut an RC in a month - target for releasing at mid of May, to give 
enough time for outstanding blocker / critical issues. Will start moving out 
any tickets that are not blockers and/or won't fit the timeline - there are 2 
blockers and 9 critical tickets outstanding as of now. For progress of 
releasing effort, please refer our release wiki [3].

   Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.0 release announcement: 
http://www.mail-archive.com/general@hadoop.apache.org/msg07443.html
[2] 2.8.1 release Blockers/Criticals: https://s.apache.org/KGxC
[3] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Upgrading minimum version of Maven to 3.1 from 3.0

[Junping Du]

Thanks Sunil to bring it up. However, any reason not to go to latest version of 
maven (3.3+)? Even version 3.1 is quite old - released 3+ years from now 
(please refer: https://archive.apache.org/dist/maven/binaries/).
btw, I assume we only plan to change maven version on trunk given new YARN UI 
only merge to trunk branch. Isn't it?

Thanks,

Junping

From: Akira Ajisaka 
Sent: Monday, April 03, 2017 10:02 PM
To: Sunil G; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: Upgrading minimum version of Maven to 3.1 from 3.0

As I mentioned in YARN-6421. I'm +1 for upgrading to 3.1+ because
the latest version of Maven 3.0.x is quite old. (4 years ago)

We need to update dev-support/docker/Dockerfile to enable Maven 3.1+ for
precommit Jenkins job.

Regards,
Akira

On 2017/04/03 18:49, Sunil G wrote:
> Hi Folks,
>
> Recently we were doing build framework up-gradation for Yarn Ui. In order
> to compile yarn-ui on various architectures, we were using
> frontend-maven-plugin 0.0.22 version.
> However build is failing in *ppc64le.* If we could use latest version of
> frontend-maven-plugin, we could resolve this error. (such as using 1.1
> version). But this requires maven version 3.1 minimum. YARN-6421 is
> tracking this issue, and we thought we can propose to upgrade to maven 3.1
>
> Kindly share your thoughts.
>
> Thanks
> + Sunil
>

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Quick update: the missing bits in svn release directories (reported in 
https://issues.apache.org/jira/browse/INFRA-13749) already get resolved with 
workaround of bypassing 200M file size limitation. We should document this in 
our release wiki.
However, a new issue is found that releasing bits are missing from mirror sites 
which get reported in https://issues.apache.org/jira/browse/INFRA-13755. The 
Apache infrastructure team is investigating right now. Will send release news 
when issue get fixed. Stay tuned!

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Friday, March 24, 2017 6:21 PM
To: Allen Wittenauer
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Thanks Allen, I already add it back in JIRA's 2.8.0 release date.
However, I met another issue in uploading our 2.8.0 release bits to SVN: 
hadoop-2.8.0.tar.gz get uploaded failed with following exception:

Adding  (bin)  hadoop-2.8.0/hadoop-2.8.0.tar.gz
Transmitting file data .svn: E175002: Commit failed (details follow):
svn: E175002: PUT request on 
'/repos/dist/!svn/txr/18902-g1s/release/hadoop/common/hadoop-2.8.0/hadoop-2.8.0.tar.gz'
 failed

Other (smaller) files get uploaded successfully. Once I suspected it could due 
to no space, so I removed some old release bits (like: 2.5.2) but still no 
luck. So may be the size of hadoop-2.8.0.tar.gz become a problem here?
I already reported the issue in 
https://issues.apache.org/jira/browse/INFRA-13749, but will appreciate if 
someone got idea on how to get through this.

Thanks,

Junping

From: Allen Wittenauer <a...@effectivemachines.com>
Sent: Thursday, March 23, 2017 5:27 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Just a heads up.  Looks like some removed the Finish Date off of 2.8.0 in JIRA. 
 It needs to be put back to match what is in the artifacts that we voted on.

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Thanks again for all who verified and voted!


I give my binding +1 to conclude the vote for 2.8.0 RC3, based on:
- Build from source and verify signatures
- Deploy pseudo-distributed cluster with capacity scheduler
- Verify UI of daemons, like: NameNode, ResourceManager, NodeManager, etc.
- Run some example MR jobs, like: PI, sleep, etc.

Now, we have:

7 binding +1s, from:
 Wangda Tan, Jason Lowe, Akira Ajisaka, Ravi Prakash,
 Karthik Kambatla, Jian He, Junping Du

18 non-binding +1s, from:
Miklos Szegedi, Eric Payne, Daniel Templeton, Mingliang Liu,
Sunil Govind, Marton Elek, Brahma Reddy Battula, Masatake Iwasaki,
Gergo Pasztor, Haibo Chen, Zhihai Xu, John Zhuge, Eric Badger,
Kuhu Shukla, Larry Mccay, Rakesh Radhakrishnan, Naganarasimha Garla,
Varun Saxena

1 binding +0, from:
Steve Loughran

and no -1s.

So I am glad to announce that the vote of 2.8.0 RC3 passes.

Thanks everyone listed above who tried the release candidate and vote.
Also, kudos to all who ever help with 2.8.0 release effort in all kinds of ways.
Without working together in community, we cannot afford so many issues get found
in RC stage (as a minor release with debit of ~2 years commits) and most of them
get fixed so quickly.

I'll push the release bits and send out an announcement for 2.8.0 soon.


Thanks,

Junping​



From: Karthik Kambatla <ka...@cloudera.com>
Sent: Wednesday, March 22, 2017 2:10 PM
To: varunsax...@apache.org
Cc: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

+1 (binding)

* Built from source
* Started a pseudo-distributed cluster with fairscheduler.
* Ran sample jobs
* Verified WebUI

On Wed, Mar 22, 2017 at 11:56 AM, 
varunsax...@apache.org<mailto:varunsax...@apache.org> 
<varun.saxena.apa...@gmail.com<mailto:varun.saxena.apa...@gmail.com>> wrote:
Thanks Junping for creating the release.

+1 (non-binding)

* Verified signatures.
* Built from source.
* Set up a pseudo-distributed cluster.
* Successfully ran pi and wordcount jobs.
* Navigated the YARN RM and NM UI.

Regards,
Varun Saxena.

On Wed, Mar 22, 2017 at 12:13 AM, Haibo Chen 
<haiboc...@cloudera.com<mailto:haiboc...@cloudera.com>> wrote:

> Thanks Junping for working on the new release!
>
> +1 non-binding
>
> 1) Downloaded the source, verified the checksum
> 2) Built natively from source, and deployed it to a pseudo-distributed
> cluster
> 3) Ran sleep and teragen job and checked both YARN and JHS web UI
> 4) Played with yarn + mapreduce command lines
>
> Best,
> Haibo Chen
>
> On Mon, Mar 20, 2017 at 11:18 AM, Junping Du 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
>
> > ?Thanks for update, John. Then we should be OK with fixing this issue in
> > 2.8.1.
> >
> > Mark the target version of HADOOP-14205 to 2.8.1 instead of 2.8.0 and
> bump
> > up to blocker in case we could miss this in releasing 2.8.1. :)
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> > 
> > From: John Zhuge <jzh...@cloudera.com<mailto:jzh...@cloudera.com>>
> > Sent: Monday, March 20, 2017 10:31 AM
> > To: Junping Du
> > Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
> > hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>;
> > yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>; 
> > mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>
> > Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >
> > Yes, it only affects ADL. There is a workaround of adding these 2
> > properties to core-site.xml:
> >
> >   
> > fs.adl.impl
> > org.apache.hadoop.fs.adl.AdlFileSystem
> >   
> >
> >   
> > fs.AbstractFileSystem.adl.impl
> > org.apache.hadoop.fs.adl.Adl
> >   
> >
> > I have the initial patch ready but hitting these live unit test failures:
> >
> > Failed tests:
> >   TestAdlFileSystemContractLive.runTest:60->FileSystemContract
> BaseTest.testListStatus:257
> > expected:<1> but was:<10>
> >
> > Tests in error:
> >   TestAdlFileContextMainOperationsLive>FileContextMainOperatio
> nsBaseTest.
> > testMkdirsFailsForSubdirectoryOfExistingFile:254 » AccessControl
> >   TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> > testMkdirsFailsForSubdirectoryOfExistingFile:190 » AccessControl
> >
> >
> > Stay tuned...
> >
> > John Zhuge
> > Software Engineer, Cloudera
> >
>

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Thanks all for response with verification work and vote!


Sounds like we are hitting several issues here, although none seems to be 
blockers so far. Given the large commit set - 2000+ commits first landed in 
branch-2 release, we may should follow 2.7.0 practice that to claim this 
release is not for production cluster, just like Vinod's suggestion in previous 
email. We should quickly come up with 2.8.1 release in next 1 or 2 month for 
production deployment.


We will close the vote in next 24 hours. For people who haven't vote, please 
keep on verification work and report any issues if founded - I will check if 
another round of RC is needed based on your findings. Thanks!


Thanks,


Junping



From: Kuhu Shukla <kshu...@yahoo-inc.com>
Sent: Tuesday, March 21, 2017 3:17 PM
Cc: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)


+1 (non-binding)

- Verified signatures.
- Downloaded and built from source tar.gz.
- Deployed a pseudo-distributed cluster on Mac Sierra.
- Ran example Sleep job successfully.
- Deployed latest Apache Tez 0.9 and ran sample Tez orderedwordcount 
successfully.

Thank you Junping and everyone else who worked on getting this release out.

Warm Regards,
Kuhu
On Tuesday, March 21, 2017, 3:42:46 PM CDT, Eric Badger 
<ebad...@yahoo-inc.com.INVALID> wrote:
+1 (non-binding)

- Verified checksums and signatures of all files
- Built from source on MacOS Sierra via JDK 1.8.0 u65
- Deployed single-node cluster
- Successfully ran a few sample jobs

Thanks,

Eric

On Tuesday, March 21, 2017 2:56 PM, John Zhuge <john.zh...@gmail.com> wrote:



+1. Thanks for the great effort, Junping!


  - Verified checksums and signatures of the tarballs
  - Built source code with Java 1.8.0_66-b17 on Mac OS X 10.12.3
  - Built source and native code with Java 1.8.0_111 on Centos 7.2.1511
  - Cloud connectors:
  - s3a: integration tests, basic fs commands
  - adl: live unit tests, basic fs commands. See notes below.
  - Deployed a pseudo cluster, passed the following sanity tests in
  both insecure and SSL mode:
  - HDFS: basic dfs, distcp, ACL commands
  - KMS and HttpFS: basic tests
  - MapReduce wordcount
  - balancer start/stop


Needs the following JIRAs to pass all ADL tests:

  - HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge.
  - HDFS-11132. Allow AccessControlException in contract tests when
  getFileStatus on subdirectory of existing files. Contributed by Vishwajeet
  Dusane
  - HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1
  runtime error. (John Zhuge via lei)


On Mon, Mar 20, 2017 at 10:31 AM, John Zhuge <jzh...@cloudera.com> wrote:

> Yes, it only affects ADL. There is a workaround of adding these 2
> properties to core-site.xml:
>
>  
>fs.adl.impl
>org.apache.hadoop.fs.adl.AdlFileSystem
>  
>
>  
>fs.AbstractFileSystem.adl.impl
>org.apache.hadoop.fs.adl.Adl
>  
>
> I have the initial patch ready but hitting these live unit test failures:
>
> Failed tests:
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testListStatus:257
> expected:<1> but was:<10>
>
> Tests in error:
>
> TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:254
> » AccessControl
>
> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> testMkdirsFailsForSubdirectoryOfExistingFile:190
> » AccessControl
>
>
> Stay tuned...
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du <j...@hortonworks.com> wrote:
>
> > Thank you for reporting the issue, John! Does this issue only affect ADL
> > (Azure Data Lake) which is a new feature for 2.8 rather than other
> existing
> > FS? If so, I think we can leave the fix to 2.8.1 to fix given this is
> not a
> > regression and just a new feature get broken.?
> >
> >
> > Thanks,
> >
> >
> > Junping
> > --
> > *From:* John Zhuge <jzh...@cloudera.com>
> > *Sent:* Monday, March 20, 2017 9:07 AM
> > *To:* Junping Du
> > *Cc:* common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> >
> > Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
> > FileSystem for scheme: adl".
> >
> > The issue were caused by backporting HADOOP-13037 to branch-2 and
> earlier.
> > HADOOP-12666 should not be backported, but so

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

?Thanks for update, John. Then we should be OK with fixing this issue in 2.8.1.

Mark the target version of HADOOP-14205 to 2.8.1 instead of 2.8.0 and bump up 
to blocker in case we could miss this in releasing 2.8.1. :)


Thanks,


Junping


From: John Zhuge <jzh...@cloudera.com>
Sent: Monday, March 20, 2017 10:31 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Yes, it only affects ADL. There is a workaround of adding these 2 properties to 
core-site.xml:

  
fs.adl.impl
org.apache.hadoop.fs.adl.AdlFileSystem
  

  
fs.AbstractFileSystem.adl.impl
org.apache.hadoop.fs.adl.Adl
  

I have the initial patch ready but hitting these live unit test failures:

Failed tests:
  
TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.testListStatus:257
 expected:<1> but was:<10>

Tests in error:
  
TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:254
 » AccessControl
  
TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:190
 » AccessControl


Stay tuned...

John Zhuge
Software Engineer, Cloudera

On Mon, Mar 20, 2017 at 10:02 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:

Thank you for reporting the issue, John! Does this issue only affect ADL (Azure 
Data Lake) which is a new feature for 2.8 rather than other existing FS? If so, 
I think we can leave the fix to 2.8.1 to fix given this is not a regression and 
just a new feature get broken.?


Thanks,


Junping


From: John Zhuge <jzh...@cloudera.com<mailto:jzh...@cloudera.com>>
Sent: Monday, March 20, 2017 9:07 AM
To: Junping Du
Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>; 
mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No FileSystem 
for scheme: adl".

The issue were caused by backporting HADOOP-13037 to branch-2 and earlier. 
HADOOP-12666 should not be backported, but some changes are needed: property 
fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.

I am working on a patch.


John Zhuge
Software Engineer, Cloudera

On Fri, Mar 17, 2017 at 2:18 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi all,
 With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Thank you for reporting the issue, John! Does this issue only affect ADL (Azure 
Data Lake) which is a new feature for 2.8 rather than other existing FS? If so, 
I think we can leave the fix to 2.8.1 to fix given this is not a regression and 
just a new feature get broken.?


Thanks,


Junping


From: John Zhuge <jzh...@cloudera.com>
Sent: Monday, March 20, 2017 9:07 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No FileSystem 
for scheme: adl".

The issue were caused by backporting HADOOP-13037 to branch-2 and earlier. 
HADOOP-12666 should not be backported, but some changes are needed: property 
fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.

I am working on a patch.


John Zhuge
Software Engineer, Cloudera

On Fri, Mar 17, 2017 at 2:18 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi all,
 With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping

[VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Junping Du]

Hi all,
 With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Junping Du]

Thanks Steve. That's Awesome! I will kick off a new RC soon.
Shall we reopen HDFS-6200 given issues here? Making it in release note of 2.8.0 
could confuse people as it doesn't work in HA deployment.

Thanks,

Junping

From: Steve Loughran
Sent: Thursday, March 16, 2017 7:27 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

> On 16 Mar 2017, at 00:25, Junping Du <j...@hortonworks.com> wrote:
>
> bq. From my read of the poms, hadoop-client depends on hadoop-hdfs-client to 
> pull in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. 
> So I think this affects users of the hadoop-client artifact, which has 
> existed for a long time.
>
> I could miss that. Thanks for reminding! From my quick check: 
> https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client/2.7.3?, it 
> sounds like 669 artifacts from other projects were depending on it.
>
>
> I think we should withdraw the current RC bits. Please stop the verification 
> & vote.
>
> I will kick off another RC immediately when HDFS-11431 get fixed.

is done. hadoop-hdfs without any server-side dependencies is now a 
hadoop-client dependency.

Release notes:

The hadoop-client POM now includes a leaner hdfs-client, stripping out all the 
transitive dependencies on JARs only needed for the Hadoop HDFS daemon itself. 
The specific jars now excluded are: leveldbjni-all, jetty-util, commons-daemon, 
xercesImpl, netty and servlet-api.

This should make downstream projects dependent JARs smaller, and avoid version 
conflict problems with the specific JARs now excluded.

Applications may encounter build problems if they did depend on these JARs, and 
which didn't explicitly include them. There are two fixes for this

* explicitly include the JARs, stating which version of them you want.
* add a dependency on hadoop-hdfs. For Hadoop 2.8+, this will add the missing 
dependencies. For builds against older versions of Hadoop, this will be 
harmless, as hadoop-hdfs and all its dependencies are already pulled in by the 
hadoop-client POM.




-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Junping Du]

bq. From my read of the poms, hadoop-client depends on hadoop-hdfs-client to 
pull in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. 
So I think this affects users of the hadoop-client artifact, which has existed 
for a long time.

I could miss that. Thanks for reminding! From my quick check: 
https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client/2.7.3?, it 
sounds like 669 artifacts from other projects were depending on it.


I think we should withdraw the current RC bits. Please stop the verification & 
vote.

I will kick off another RC immediately when HDFS-11431 get fixed.


Thanks,


Junping



From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Wednesday, March 15, 2017 2:04 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

Hi Junping, inline,


>From my understanding, this issue is related to our previous improvements with 
>separating client and server jars in HDFS-6200. If we use the new "client" jar 
>in NN HA deployment, then we will hit the issue reported.

>From my read of the poms, hadoop-client depends on hadoop-hdfs-client to pull 
>in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. So I 
>think this affects users of the hadoop-client artifact, which has existed for 
>a long time.

Essentially all of our customer deployments run with NN HA, so this would 
affect a lot of users.

I can see two options here:

- Without any change in 2.8.0, if user hit the issue when they deploy HA 
cluster by using new client jar, adding back hdfs jar just like how things work 
previously

- Make the change now in 2.8.0, either moving ConfiguredFailoverProxyProvider 
to client jar or adding dependency between client jar and server jar. There 
must be some arguments there on which way to fix is better especially 
ConfiguredFailoverProxyProvider still has some sever side dependencies.


I would prefer the first option, given:

- The issue fixing time is unpredictable as there are still discussion on how 
to fix this issue. Our 2.8.0 release shouldn't be an endless journey which has 
been deferred several times for more serious issue.

Looks like we have a patch being actively revved and reviewed to fix this by 
making hadoop-hdfs-client depend on hadoop-hdfs. Thanks to Steven and Steve for 
working on this.

Steve proposed doing a proper split in a later JIRA.

- We have workaround for this improvement, no regression happens due to this 
issue. People can still use hdfs jar in old way. The worst case is improvement 
for HDFS doesn't work in some cases - that shouldn't block the whole release.

Based on the above, I think there is a regression for users of the 
hadoop-client artifact.

If it actually only affects users of hadoop-hdfs-client, then I agree we can 
document it as a Known Issue and fix it later.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Junping Du]

Thanks Andrew for reporting the issue. This JIRA is out of my radar as it? 
didn't specify any target version before.


>From my understanding, this issue is related to our previous improvements with 
>separating client and server jars in HDFS-6200. If we use the new "client" jar 
>in NN HA deployment, then we will hit the issue reported.


I can see two options here:

- Without any change in 2.8.0, if user hit the issue when they deploy HA 
cluster by using new client jar, adding back hdfs jar just like how things work 
previously

- Make the change now in 2.8.0, either moving ConfiguredFailoverProxyProvider 
to client jar or adding dependency between client jar and server jar. There 
must be some arguments there on which way to fix is better especially 
ConfiguredFailoverProxyProvider still has some sever side dependencies.


I would prefer the first option, given:

- The issue fixing time is unpredictable as there are still discussion on how 
to fix this issue. Our 2.8.0 release shouldn't be an endless journey which has 
been deferred several times for more serious issue.

- We have workaround for this improvement, no regression happens due to this 
issue. People can still use hdfs jar in old way. The worst case is improvement 
for HDFS doesn't work in some cases - that shouldn't block the whole release.


I think we should let vote keep going unless someone have more concerns which I 
could miss.



Thanks,


Junping



From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Tuesday, March 14, 2017 2:50 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

Hi Junping,

Noticed this possible blocker float by my inbox today. It had an affects but no 
target version set:

https://issues.apache.org/jira/browse/HDFS-11431

Thoughts? Seems like the hadoop-hdfs-client artifact doesn't work right now.

Best,
Andrew


On Tue, Mar 14, 2017 at 1:41 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi all,
 With several important fixes get merged last week, I've created a new 
release candidate (RC2) for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,919 fixes, improvements, and new features. 
Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  Please note that RC0 and RC1 are not voted public because significant 
issues are found just after RC tag getting published.

  The RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC2

  The RC tag in git is: release-2.8.0-RC2

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1056

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/20/2017 PDT time.

Thanks,

Junping

[VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Junping Du]

Hi all,
 With several important fixes get merged last week, I've created a new 
release candidate (RC2) for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,919 fixes, improvements, and new features. 
Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  Please note that RC0 and RC1 are not voted public because significant 
issues are found just after RC tag getting published.

  The RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC2

  The RC tag in git is: release-2.8.0-RC2

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1056

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/20/2017 PDT time.

Thanks,

Junping

[jira] [Created] (MAPREDUCE-6852) Job#updateStatus() failed with NPE due to race condition

[Junping Du (JIRA)]

Junping Du created MAPREDUCE-6852:
-

 Summary: Job#updateStatus() failed with NPE due to race condition
 Key: MAPREDUCE-6852
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6852
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Junping Du
Assignee: Junping Du


Like MAPREDUCE-6762, we found this issue in a cluster where Pig query 
occasionally failed on NPE - "Pig uses JobControl API to track MR job status, 
but sometimes Job History Server failed to flush job meta files to HDFS which 
caused the status update failed." Beside NPE in o.a.h.mapreduce.Job.getJobName, 
we also get NPE in Job.updateStatus() and the exception is as following:
{noformat}
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:323)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:320)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1833)
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:604)
{noformat}
We found state here is null. However, we already check the job state to be 
RUNNING as code below:
{noformat}
  public boolean isComplete() throws IOException {
ensureState(JobState.RUNNING);
updateStatus();
return status.isJobComplete();
  }
{noformat}
The only possible reason here is two threads are calling here for the same 
time: ensure state first, then one thread update the state to null while the 
other thread hit NPE issue here.
We should fix this NPE exception.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Yes. I did maven deploy in root directory before close the staging repository. 
If this is the only suspect, I can drop the repository and do mvn deploy again.


Thanks,


Junping


From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Friday, January 20, 2017 10:48 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

You can check the error message by clicking on it, a bunch like this:

Missing Signature: 
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.8.0/hadoop-mapreduce-client-jobclient-2.8.0-tests.jar.asc'
 does not exist for 'hadoop-mapreduce-client-jobclient-2.8.0-tests.jar'.

Did you maven deploy all the signature files?

On Fri, Jan 20, 2017 at 2:04 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi,
I have successfully built the release bit on branch-2.8.0 by following: 
https://wiki.apache.org/hadoop/HowToRelease step by step. However, when try to 
close the "Staging Repositories" at Nexus page 
(https://repository.apache.org/#stagingRepositories), I found our repository - 
orgapachehadoop-1051 cannot be closed due to some signature validation failed 
and one Apache rule failed. I never met this problem before in previous 
releasing 2.6.3 and 2.6.4. Is this related to our recent changes on release 
process/tools (docker based)? Any ideas or thoughts on how to fix the problem?

Thanks,

Junping

____________
From: Junping Du <j...@hortonworks.com<mailto:j...@hortonworks.com>>
Sent: Thursday, January 19, 2017 6:46 PM
To: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>
Cc: Varun Vasudev
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

According to Varun's offline email, the security fixes has landed on branch-2, 
2.8 and 2.8.0 branch.
I was kicking off a new RC build (RC1), and will publish it for vote soon. In 
the mean time, please mark fix version as 2.8.1 for any new commits landed on 
branch-2.8, and don't commit anything to branch-2.8.0 at this moment. Thanks!

Cheers,

Junping


From: Junping Du <j...@hortonworks.com<mailto:j...@hortonworks.com>>
Sent: Wednesday, January 18, 2017 3:26 PM
To: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>
Cc: Varun Vasudev
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi folks,
 In the passed one or two weeks, we found some new blockers get coming on 
branch-2.8, like: YARN-6068 (log aggregation get stuck when NM restart with 
work preserving enabled) and YARN-6072 (RM unable to start in secure mode). 
Both of them are fixed now (YARN-6072 is fixed by Ajith and I fixed YARN-6068), 
and I was starting the RC build process since early this week. As we have 
significant build tools/process change (docker based) since 2.8 comparing with 
2.6/2.7, it takes me a while to get familiar with it and finally get a 
successful build on 2.8.0-RC0 last night.
 I already push RC0 tag into public which is prerequisite step before RC 
voting. However,  in the mean while, I was pinged by Varun Vasudev that there 
are a known vulnerability issues on container_executor get identified and 
discussed in hadoop security email threads - looks like YARN-5704 fixed part of 
it, but left part - the privilege escalation via /proc/self/environ is not 
fixed yet. So most likely, I have to withdraw our 2.8.0 RC0 although I haven't 
announced it public for vote yet. I will wait this issue get fixed to prepare a 
new release candidate. As RC0 tag cannot be reverted after push into apache, 
our next release candidate will start from RC1.
As I mentioned in early email, 2.8.0 is a very big release (2300+ commits 
since 2.7.3) and I am glad that we are almost there. Thanks everyone for being 
patient and contributing our release work. Please let me know if you have more 
comments or suggestions.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com<mailto:j...@hortonworks.com>>
Sent: Wednesday, January 04, 2017 1:41 AM
To: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>; 
yarn-...@hadoop.apache.o

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Hi,
I have successfully built the release bit on branch-2.8.0 by following: 
https://wiki.apache.org/hadoop/HowToRelease step by step. However, when try to 
close the "Staging Repositories" at Nexus page 
(https://repository.apache.org/#stagingRepositories), I found our repository - 
orgapachehadoop-1051 cannot be closed due to some signature validation failed 
and one Apache rule failed. I never met this problem before in previous 
releasing 2.6.3 and 2.6.4. Is this related to our recent changes on release 
process/tools (docker based)? Any ideas or thoughts on how to fix the problem?

Thanks,

Junping

____
From: Junping Du <j...@hortonworks.com>
Sent: Thursday, January 19, 2017 6:46 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: Varun Vasudev
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

According to Varun's offline email, the security fixes has landed on branch-2, 
2.8 and 2.8.0 branch.
I was kicking off a new RC build (RC1), and will publish it for vote soon. In 
the mean time, please mark fix version as 2.8.1 for any new commits landed on 
branch-2.8, and don't commit anything to branch-2.8.0 at this moment. Thanks!

Cheers,

Junping

________
From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, January 18, 2017 3:26 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: Varun Vasudev
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi folks,
 In the passed one or two weeks, we found some new blockers get coming on 
branch-2.8, like: YARN-6068 (log aggregation get stuck when NM restart with 
work preserving enabled) and YARN-6072 (RM unable to start in secure mode). 
Both of them are fixed now (YARN-6072 is fixed by Ajith and I fixed YARN-6068), 
and I was starting the RC build process since early this week. As we have 
significant build tools/process change (docker based) since 2.8 comparing with 
2.6/2.7, it takes me a while to get familiar with it and finally get a 
successful build on 2.8.0-RC0 last night.
 I already push RC0 tag into public which is prerequisite step before RC 
voting. However,  in the mean while, I was pinged by Varun Vasudev that there 
are a known vulnerability issues on container_executor get identified and 
discussed in hadoop security email threads - looks like YARN-5704 fixed part of 
it, but left part - the privilege escalation via /proc/self/environ is not 
fixed yet. So most likely, I have to withdraw our 2.8.0 RC0 although I haven't 
announced it public for vote yet. I will wait this issue get fixed to prepare a 
new release candidate. As RC0 tag cannot be reverted after push into apache, 
our next release candidate will start from RC1.
As I mentioned in early email, 2.8.0 is a very big release (2300+ commits 
since 2.7.3) and I am glad that we are almost there. Thanks everyone for being 
patient and contributing our release work. Please let me know if you have more 
comments or suggestions.

Thanks,

Junping
________
From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, January 04, 2017 1:41 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi all Hadoopers,
 I just commit YARN-3866 which is the last blocker for branch-2.8, so since 
tomorrow I will kick off process to prepare the first RC for our 2.8.0 release. 
This release will include at least 2374 commits that never landed before in 
previous 2.x releases. Please check https://s.apache.org/RW5k for details. 
There are also 352 fixes marked in 2.7.1, 2.7.2 and 2.7.3 but not marked with 
2.8 (https://s.apache.org/RKti) that I need to double check all are landed in 
branch-2.8 and fix the versions before kicking off the first RC.
 Our progress is slightly behind my estimation weeks ago. However, 
considering we just go through holidays and Jenkins cleanup issues linger on 
for several projects (YARN, etc.), our achievement here is still great! Thanks 
everyone for keep calm and carry on the help for the release. Kudos to Wangda, 
Jian, Akira, Jason, Sangjin, Karthik and all for pushing hard on blockers 
during this time. Also, special thanks to Vinod and Andrew for sharing 
knowledge and practice (include scripts for auto version check) for releasing 
effort.
 Will try to prepare RC0 of 2.8 release for vote within this week. Stay 
tuned!

Thanks,

Junping

____________
From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, December 07, 2016 11:31 AM
To: Akira Ajisaka; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

According to Varun's offline email, the security fixes has landed on branch-2, 
2.8 and 2.8.0 branch. 
I was kicking off a new RC build (RC1), and will publish it for vote soon. In 
the mean time, please mark fix version as 2.8.1 for any new commits landed on 
branch-2.8, and don't commit anything to branch-2.8.0 at this moment. Thanks!

Cheers,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, January 18, 2017 3:26 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: Varun Vasudev
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi folks,
 In the passed one or two weeks, we found some new blockers get coming on 
branch-2.8, like: YARN-6068 (log aggregation get stuck when NM restart with 
work preserving enabled) and YARN-6072 (RM unable to start in secure mode). 
Both of them are fixed now (YARN-6072 is fixed by Ajith and I fixed YARN-6068), 
and I was starting the RC build process since early this week. As we have 
significant build tools/process change (docker based) since 2.8 comparing with 
2.6/2.7, it takes me a while to get familiar with it and finally get a 
successful build on 2.8.0-RC0 last night.
 I already push RC0 tag into public which is prerequisite step before RC 
voting. However,  in the mean while, I was pinged by Varun Vasudev that there 
are a known vulnerability issues on container_executor get identified and 
discussed in hadoop security email threads - looks like YARN-5704 fixed part of 
it, but left part - the privilege escalation via /proc/self/environ is not 
fixed yet. So most likely, I have to withdraw our 2.8.0 RC0 although I haven't 
announced it public for vote yet. I will wait this issue get fixed to prepare a 
new release candidate. As RC0 tag cannot be reverted after push into apache, 
our next release candidate will start from RC1.
As I mentioned in early email, 2.8.0 is a very big release (2300+ commits 
since 2.7.3) and I am glad that we are almost there. Thanks everyone for being 
patient and contributing our release work. Please let me know if you have more 
comments or suggestions.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, January 04, 2017 1:41 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi all Hadoopers,
 I just commit YARN-3866 which is the last blocker for branch-2.8, so since 
tomorrow I will kick off process to prepare the first RC for our 2.8.0 release. 
This release will include at least 2374 commits that never landed before in 
previous 2.x releases. Please check https://s.apache.org/RW5k for details. 
There are also 352 fixes marked in 2.7.1, 2.7.2 and 2.7.3 but not marked with 
2.8 (https://s.apache.org/RKti) that I need to double check all are landed in 
branch-2.8 and fix the versions before kicking off the first RC.
 Our progress is slightly behind my estimation weeks ago. However, 
considering we just go through holidays and Jenkins cleanup issues linger on 
for several projects (YARN, etc.), our achievement here is still great! Thanks 
everyone for keep calm and carry on the help for the release. Kudos to Wangda, 
Jian, Akira, Jason, Sangjin, Karthik and all for pushing hard on blockers 
during this time. Also, special thanks to Vinod and Andrew for sharing 
knowledge and practice (include scripts for auto version check) for releasing 
effort.
 Will try to prepare RC0 of 2.8 release for vote within this week. Stay 
tuned!

Thanks,

Junping

____
From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, December 07, 2016 11:31 AM
To: Akira Ajisaka; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for 
several runs before last weekend, but it get failed for latest several runs, 
probably affected by recently Jenkins down.
However, from my recent manual kick off, it seems to be good again: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I will 
keep an eye for today's nightly run to see if it back to normal.


Thanks,

Junping


From: Akira Ajisaka <ajisa...@apache.org>
Sent: Wednesday, December 07, 2016 12:12 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Junping and Andrew!

HADOOP-2.8-JACC is not working well, so I manually kicked a job to
compare 2.8 with 2.7.3.

https://builds.

Re: [VOTE] Release cadence and EOL

[Junping Du]

+1 on Sangjin's proposal - 
"A minor release line is end-of-lifed 2 years after it is released or there
are 2 newer minor releases, whichever is sooner. The community reserves the
right to extend or shorten the life of a release line if there is a good
reason to do so."

I also noticed Karthik bring up some new proposals - some of them looks 
interesting to me and I have some ideas as well. Karthik, can you bring it out 
in a separated discussion threads so that we can discuss from there?

About Chris Trezzo's question about definition of EOL of hadoop release, I 
think potentially changes could be: 
1. For users of Apache hadoop, they would expect to upgrade to a new 
minor/major releases after EOL of their current release because there is no 
guarantee of new maintenance release.

2. For release effort, apache law claim that committer can volunteer RM for any 
release. With this release EOL proposal passes and written into hadoop bylaw, 
anyone want to call for a release which is EOL then she/he have to provide a 
good reason to community and get voted before to start release effort. We don't 
want to waste community time/resource to verify/vote a narrow interested 
release.

3. About committer's responsibility, I think the bottom line is committer 
should commit patch contributor's target release and her/his own interest 
release which I conservatively agree with Allen's point that this vote doesn't 
change anything. But if a committer want to take care more interest from the 
whole community like most committers are doing today, he/she should understand 
which branches can benefit more people and could skip some EOL release branches 
for backport effort.

About major release EOL, this could be more complicated and I think we should 
discuss separately.

Thanks,

Junping

From: Allen Wittenauer 
Sent: Wednesday, January 18, 2017 3:30 PM
To: Chris Trezzo
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release cadence and EOL

> On Jan 18, 2017, at 11:21 AM, Chris Trezzo  wrote:
>
> Thanks Sangjin for pushing this forward! I have a few questions:

These are great questions, because I know I'm not seeing a whole lot of 
substance in this vote.  The way to EOL software in the open source universe is 
with new releases and aging it out.  If someone wants to be a RE for a new 
branch-1 release, more power to them.  As volunteers to the ASF, we're not on 
the hook to provide much actual support.  This feels more like a vendor play 
than a community one.  But if the PMC want to vote on it, whatever.  It won't 
be first bylaw that doesn't really mean much.

> 1. What is the definition of end-of-life for a release in the hadoop
> project? My current understanding is as follows: When a release line
> reaches end-of-life, there are no more planned releases for that line.
> Committers are no longer responsible for back-porting bug fixes to the line
> (including fixed security vulnerabilities) and it is essentially
> unmaintained.

Just a point of clarification.  There is no policy that says that 
committers must back port.  It's up to the individual committers to push a 
change onto any particular branch. Therefore, this vote doesn't really change 
anything in terms of committer responsibilities here.

> 2. How do major releases affect the end-of-life proposal? For example, how
> does a new minor release in the next major release affect the end-of-life
> of minor releases in a previous major release? Is it possible to have a
> maintained 2.x release if there is a 3.3 release?

I'm looking forward to seeing this answer too, given that 2.7.0 is 
probably past the 2 year mark, 2.8.0 has seemingly been in a holding pattern 
for over a year, and the next 3.0.0 alpha should be RSN

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Hi folks,
 In the passed one or two weeks, we found some new blockers get coming on 
branch-2.8, like: YARN-6068 (log aggregation get stuck when NM restart with 
work preserving enabled) and YARN-6072 (RM unable to start in secure mode). 
Both of them are fixed now (YARN-6072 is fixed by Ajith and I fixed YARN-6068), 
and I was starting the RC build process since early this week. As we have 
significant build tools/process change (docker based) since 2.8 comparing with 
2.6/2.7, it takes me a while to get familiar with it and finally get a 
successful build on 2.8.0-RC0 last night. 
 I already push RC0 tag into public which is prerequisite step before RC 
voting. However,  in the mean while, I was pinged by Varun Vasudev that there 
are a known vulnerability issues on container_executor get identified and 
discussed in hadoop security email threads - looks like YARN-5704 fixed part of 
it, but left part - the privilege escalation via /proc/self/environ is not 
fixed yet. So most likely, I have to withdraw our 2.8.0 RC0 although I haven't 
announced it public for vote yet. I will wait this issue get fixed to prepare a 
new release candidate. As RC0 tag cannot be reverted after push into apache, 
our next release candidate will start from RC1.
As I mentioned in early email, 2.8.0 is a very big release (2300+ commits 
since 2.7.3) and I am glad that we are almost there. Thanks everyone for being 
patient and contributing our release work. Please let me know if you have more 
comments or suggestions.

Thanks,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, January 04, 2017 1:41 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Hi all Hadoopers,
 I just commit YARN-3866 which is the last blocker for branch-2.8, so since 
tomorrow I will kick off process to prepare the first RC for our 2.8.0 release. 
This release will include at least 2374 commits that never landed before in 
previous 2.x releases. Please check https://s.apache.org/RW5k for details. 
There are also 352 fixes marked in 2.7.1, 2.7.2 and 2.7.3 but not marked with 
2.8 (https://s.apache.org/RKti) that I need to double check all are landed in 
branch-2.8 and fix the versions before kicking off the first RC.
 Our progress is slightly behind my estimation weeks ago. However, 
considering we just go through holidays and Jenkins cleanup issues linger on 
for several projects (YARN, etc.), our achievement here is still great! Thanks 
everyone for keep calm and carry on the help for the release. Kudos to Wangda, 
Jian, Akira, Jason, Sangjin, Karthik and all for pushing hard on blockers 
during this time. Also, special thanks to Vinod and Andrew for sharing 
knowledge and practice (include scripts for auto version check) for releasing 
effort.
 Will try to prepare RC0 of 2.8 release for vote within this week. Stay 
tuned!

Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, December 07, 2016 11:31 AM
To: Akira Ajisaka; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for 
several runs before last weekend, but it get failed for latest several runs, 
probably affected by recently Jenkins down.
However, from my recent manual kick off, it seems to be good again: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I will 
keep an eye for today's nightly run to see if it back to normal.


Thanks,

Junping


From: Akira Ajisaka <ajisa...@apache.org>
Sent: Wednesday, December 07, 2016 12:12 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Junping and Andrew!

HADOOP-2.8-JACC is not working well, so I manually kicked a job to
compare 2.8 with 2.7.3.

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/24/artifact/target/compat-check/report.html

Regards,
Akira

On 2016/12/02 2:08, Junping Du wrote:
> Thanks Andrew! That's also a nice suggestion. I already create a similar job: 
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/ for 2.8 
> and kick off several runs manually. Will monitor incompatible status from 
> there.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Andrew Wang <andrew.w...@cloudera.com>
> Sent: Wednesday, November 30, 2016 4:18 PM
> To: Junping Du
> Cc: Sangjin Lee; common-...@hadoop.apache.org; hdfs-...@hadoop

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Hi all Hadoopers,
 I just commit YARN-3866 which is the last blocker for branch-2.8, so since 
tomorrow I will kick off process to prepare the first RC for our 2.8.0 release. 
This release will include at least 2374 commits that never landed before in 
previous 2.x releases. Please check https://s.apache.org/RW5k for details. 
There are also 352 fixes marked in 2.7.1, 2.7.2 and 2.7.3 but not marked with 
2.8 (https://s.apache.org/RKti) that I need to double check all are landed in 
branch-2.8 and fix the versions before kicking off the first RC.
 Our progress is slightly behind my estimation weeks ago. However, 
considering we just go through holidays and Jenkins cleanup issues linger on 
for several projects (YARN, etc.), our achievement here is still great! Thanks 
everyone for keep calm and carry on the help for the release. Kudos to Wangda, 
Jian, Akira, Jason, Sangjin, Karthik and all for pushing hard on blockers 
during this time. Also, special thanks to Vinod and Andrew for sharing 
knowledge and practice (include scripts for auto version check) for releasing 
effort.
 Will try to prepare RC0 of 2.8 release for vote within this week. Stay 
tuned!

Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Wednesday, December 07, 2016 11:31 AM
To: Akira Ajisaka; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for 
several runs before last weekend, but it get failed for latest several runs, 
probably affected by recently Jenkins down.
However, from my recent manual kick off, it seems to be good again: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I will 
keep an eye for today's nightly run to see if it back to normal.


Thanks,

Junping


From: Akira Ajisaka <ajisa...@apache.org>
Sent: Wednesday, December 07, 2016 12:12 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Junping and Andrew!

HADOOP-2.8-JACC is not working well, so I manually kicked a job to
compare 2.8 with 2.7.3.

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/24/artifact/target/compat-check/report.html

Regards,
Akira

On 2016/12/02 2:08, Junping Du wrote:
> Thanks Andrew! That's also a nice suggestion. I already create a similar job: 
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/ for 2.8 
> and kick off several runs manually. Will monitor incompatible status from 
> there.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Andrew Wang <andrew.w...@cloudera.com>
> Sent: Wednesday, November 30, 2016 4:18 PM
> To: Junping Du
> Cc: Sangjin Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Vinod 
> Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> I recommend giving the JACC report another look. I set up a parameterized 
> jenkins job which you can trigger manually for branch-2.8:
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/
>
> On Wed, Nov 30, 2016 at 4:06 PM, Junping Du 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
> Hi Sangjin and all,
>
>  That sounds good. If anyone have priority fix to backport , please 
> nominate it by following this email thread or ping me on JIRA directly. And I 
> agree that we should keep in mind that our bar for 2.8 release should be high 
> now as we want to move quickly on this release. Non-critical fixes can wait 
> for next one.
>
>   Any other comments and suggestions?
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: sjl...@gmail.com<mailto:sjl...@gmail.com> 
> <sjl...@gmail.com<mailto:sjl...@gmail.com>> on behalf of Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org>>
> Sent: Wednesday, November 30, 2016 3:24 PM
> To: Junping Du
> Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
> hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
> mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>; 
> yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>; Vinod 
> Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com<mailto:sj...@twitter.com>
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> T

YARN Jenkins Build get consistent failed.

[Junping Du]

Hi hadoop folks,

   I noticed that our recent YARN jenkins tests are consistently failed 
(https://builds.apache.org/job/PreCommit-YARN-Build) due to test environment 
issues below.

   I already filed blocker issue 
https://issues.apache.org/jira/browse/INFRA-13141 to our INFRA team yesterday 
but haven't get any response yet. All commit work on YARN project are fully 
blocked. Anyone have ideas on how to move things forward?

btw, Jenkins tests for hadoop/hdfs/mapreduce seems to be OK.


FATAL: Command "git clean -fdx" returned status code 1:
stdout:
stderr: warning: failed to remove 
hadoop-common-project/hadoop-common/target/test/data/3

hudson.plugins.git.GitException:
 Command "git clean -fdx" returned status code 1:
stdout:
stderr: warning: failed to remove 
hadoop-common-project/hadoop-common/target/test/data/3

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1699)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1695)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1317)




Thanks,


Junping

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for 
several runs before last weekend, but it get failed for latest several runs, 
probably affected by recently Jenkins down. 
However, from my recent manual kick off, it seems to be good again: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I will 
keep an eye for today's nightly run to see if it back to normal.


Thanks,

Junping


From: Akira Ajisaka <ajisa...@apache.org>
Sent: Wednesday, December 07, 2016 12:12 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Junping and Andrew!

HADOOP-2.8-JACC is not working well, so I manually kicked a job to
compare 2.8 with 2.7.3.

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/24/artifact/target/compat-check/report.html

Regards,
Akira

On 2016/12/02 2:08, Junping Du wrote:
> Thanks Andrew! That's also a nice suggestion. I already create a similar job: 
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/ for 2.8 
> and kick off several runs manually. Will monitor incompatible status from 
> there.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Andrew Wang <andrew.w...@cloudera.com>
> Sent: Wednesday, November 30, 2016 4:18 PM
> To: Junping Du
> Cc: Sangjin Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Vinod 
> Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> I recommend giving the JACC report another look. I set up a parameterized 
> jenkins job which you can trigger manually for branch-2.8:
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/
>
> On Wed, Nov 30, 2016 at 4:06 PM, Junping Du 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
> Hi Sangjin and all,
>
>  That sounds good. If anyone have priority fix to backport , please 
> nominate it by following this email thread or ping me on JIRA directly. And I 
> agree that we should keep in mind that our bar for 2.8 release should be high 
> now as we want to move quickly on this release. Non-critical fixes can wait 
> for next one.
>
>   Any other comments and suggestions?
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: sjl...@gmail.com<mailto:sjl...@gmail.com> 
> <sjl...@gmail.com<mailto:sjl...@gmail.com>> on behalf of Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org>>
> Sent: Wednesday, November 30, 2016 3:24 PM
> To: Junping Du
> Cc: common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
> hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
> mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>; 
> yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>; Vinod 
> Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com<mailto:sj...@twitter.com>
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> Thanks for picking this up Junping!
>
> I heard on email threads and offline discussions that there may be interests 
> in porting some fixes from branch-2 to branch-2.8. I still feel that the bar 
> should be pretty high to do that, but it might be good to have folks suggest 
> some critical fixes that need to be released sooner. Thoughts?
>
> On Tue, Nov 29, 2016 at 8:50 PM, Junping Du 
> <j...@hortonworks.com<mailto:j...@hortonworks.com><mailto:j...@hortonworks.com<mailto:j...@hortonworks.com>>>
>  wrote:
> Hi folks,
>  Per asked by Vinod offline, I would like to continue the effort to push 
> 2.8 release out in next several weeks. The plan is as following:
>  - We only have 5 blockers and 3 critical issues 
> (https://s.apache.org/6kwx) so far, and most of them are in good progress 
> now. I put them all on apache ciwiki ( 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release) and 
> will do daily monitor and update on these issues in next couple of weeks.
>  - When all blocker/critical issues are gone (target EOD is Dec. 10), I 
> will push out all major/minor issues that is open. So if you are working on 
> some major issues and target for release in 2.8, please keep track of our 
> release status and your work progress.
>  - I will create RC immediately once our 2.8 target issues are clean 
> (especially for blocker and critical issues) for voting process.
>
>  I hope we

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Hi Sangjin and all,

 That sounds good. If anyone have priority fix to backport , please 
nominate it by following this email thread or ping me on JIRA directly. And I 
agree that we should keep in mind that our bar for 2.8 release should be high 
now as we want to move quickly on this release. Non-critical fixes can wait for 
next one.

  Any other comments and suggestions?


Thanks,


Junping



From: sjl...@gmail.com <sjl...@gmail.com> on behalf of Sangjin Lee 
<sj...@apache.org>
Sent: Wednesday, November 30, 2016 3:24 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Vinod Vavilapalli; 
Jian He; Wangda Tan; sj...@twitter.com
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks for picking this up Junping!

I heard on email threads and offline discussions that there may be interests in 
porting some fixes from branch-2 to branch-2.8. I still feel that the bar 
should be pretty high to do that, but it might be good to have folks suggest 
some critical fixes that need to be released sooner. Thoughts?

On Tue, Nov 29, 2016 at 8:50 PM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi folks,
 Per asked by Vinod offline, I would like to continue the effort to push 
2.8 release out in next several weeks. The plan is as following:
 - We only have 5 blockers and 3 critical issues 
(https://s.apache.org/6kwx) so far, and most of them are in good progress now. 
I put them all on apache ciwiki ( 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release) and will 
do daily monitor and update on these issues in next couple of weeks.
 - When all blocker/critical issues are gone (target EOD is Dec. 10), I 
will push out all major/minor issues that is open. So if you are working on 
some major issues and target for release in 2.8, please keep track of our 
release status and your work progress.
 - I will create RC immediately once our 2.8 target issues are clean 
(especially for blocker and critical issues) for voting process.

 I hope we can make it within year 2016, best before Xmas holiday. And I 
need help from all of you. :)


Thanks,

Junping


From: Vinod Kumar Vavilapalli <vino...@apache.org<mailto:vino...@apache.org>>
Sent: Monday, July 25, 2016 2:57 PM
To: Jian He
Cc: mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>
Subject: Re: [Release thread] 2.8.0 release activities

Thanks for all that great help moving this release forward, Jian, Sangjin, 
Wangda et al! Appreciate it.

The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last week.

I only see 9 blocker / critical fixes pending for 2.8.0 - 
https://issues.apache.org/jira/issues/?filter=12334985. Let’s do this!

Thanks
+Vinod

> On May 11, 2016, at 6:04 PM, Jian He 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
>
> For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs attention 
> for 2.8
>
> MAPREDUCE-6288
> YARN-1815
> YARN-4685
> YARN-4844
>
> The rest are either improvements or long-standing issues and does not qualify 
> release blocker, IMO.
> I think we’ll try to get these 4 jiras in asap. The rest will be on best 
> effort, resolve as much as possible and move them out if not resolved in time.
>
> Jian
>
> On May 11, 2016, at 5:37 PM, Wangda Tan 
> <wheele...@gmail.com<mailto:wheele...@gmail.com><mailto:wheele...@gmail.com<mailto:wheele...@gmail.com>>>
>  wrote:
>
> Sounds good to me :).
>
> Jian and I have looked at all existing 2.8.0 blockers and criticals today.
> To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
> out. Left comments on these JIRAs asked original owners, plan to update
> target version of these JIRAs early next week.
>
> Will keep this thread updated.
>
> Thanks,
> Wangda
>
>
> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org><mailto:sj...@apache.org<mailto:sj...@apache.org>>>
>  wrote:
>
> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
> report the list here. Then folks can chime in if you agree
>
> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
>
> Thanks,
> Sangjin
>
> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan 
> <wheele...@gmail.com<mailto:wheele...@gmail.com

[Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Hi folks,
 Per asked by Vinod offline, I would like to continue the effort to push 
2.8 release out in next several weeks. The plan is as following:
 - We only have 5 blockers and 3 critical issues 
(https://s.apache.org/6kwx) so far, and most of them are in good progress now. 
I put them all on apache ciwiki ( 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release) and will 
do daily monitor and update on these issues in next couple of weeks.
 - When all blocker/critical issues are gone (target EOD is Dec. 10), I 
will push out all major/minor issues that is open. So if you are working on 
some major issues and target for release in 2.8, please keep track of our 
release status and your work progress.
 - I will create RC immediately once our 2.8 target issues are clean 
(especially for blocker and critical issues) for voting process. 

 I hope we can make it within year 2016, best before Xmas holiday. And I 
need help from all of you. :)


Thanks,

Junping


From: Vinod Kumar Vavilapalli 
Sent: Monday, July 25, 2016 2:57 PM
To: Jian He
Cc: mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; common-...@hadoop.apache.org
Subject: Re: [Release thread] 2.8.0 release activities

Thanks for all that great help moving this release forward, Jian, Sangjin, 
Wangda et al! Appreciate it.

The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last week.

I only see 9 blocker / critical fixes pending for 2.8.0 - 
https://issues.apache.org/jira/issues/?filter=12334985. Let’s do this!

Thanks
+Vinod

> On May 11, 2016, at 6:04 PM, Jian He  wrote:
>
> For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs attention 
> for 2.8
>
> MAPREDUCE-6288
> YARN-1815
> YARN-4685
> YARN-4844
>
> The rest are either improvements or long-standing issues and does not qualify 
> release blocker, IMO.
> I think we’ll try to get these 4 jiras in asap. The rest will be on best 
> effort, resolve as much as possible and move them out if not resolved in time.
>
> Jian
>
> On May 11, 2016, at 5:37 PM, Wangda Tan 
> > wrote:
>
> Sounds good to me :).
>
> Jian and I have looked at all existing 2.8.0 blockers and criticals today.
> To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
> out. Left comments on these JIRAs asked original owners, plan to update
> target version of these JIRAs early next week.
>
> Will keep this thread updated.
>
> Thanks,
> Wangda
>
>
> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee 
> > wrote:
>
> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
> report the list here. Then folks can chime in if you agree
>
> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
>
> Thanks,
> Sangjin
>
> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan 
> > wrote:
>
> +1, we should close such staled JIRAs to avoid doing unnecessary checks
> for
> every releases.
>
> I'm working on reviewing YARN/MR critical/blocker patches currently, it
> gonna very helpful if someone else can help with reviewing Common/HDFS
> JIRAs.
>
> Thanks,
> Wangda
>
>
> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee 
> > wrote:
>
> Where do we stand in terms of closing out blocker/critical issues for
> 2.8.0? I still see 50 open JIRAs in Vinod's list:
> https://issues.apache.org/jira/issues/?filter=12334985
>
> But I see a lot of JIRAs with no patches or very stale patches. It
> would be
> a good exercise to come up with the list of JIRAs that we need to block
> 2.8.0 for and focus our attention on closing them out. Thoughts?
>
> Thanks,
> Sangjin
>
> On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 
> 
>
> wrote:
>
>
> On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> We are not converging - there’s still 58 more. I need help from the
> community in addressing / review 2.8.0 blockers. If folks can start
> with
> reviewing Patch available tickets, that’ll be great.
>
>
>
>
> I'm still doing the s3a stuff, other people testing and reviewing this
> stuff welcome.
>
> in particular, I could do with others playing with this patch of mine,
> which adds counters and things into S3a, based on the azure
> instrumentation
>
> https://issues.apache.org/jira/browse/HADOOP-13028
>
>
>
>
>
>
>
>


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To 

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Junping Du]

+1 binding, based on:

- Download and build from source, check signatures

- Deploy a single node cluster and run some simple jobs, like sleep, PI, etc.

Thanks Sangjin and Chris for working on 2.6.5 release!

Thanks,

Junping

From: larry mccay 
Sent: Saturday, October 08, 2016 3:43 AM
To: Karthik Kambatla
Cc: Yongjun Zhang; Sangjin Lee; Andrew Wang; Wangda Tan; Zhihai Xu; Masatake 
Iwasaki; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

+1 (non-binding)


* Downloaded and verified signatures

* Built from source

* Deployed a standalone cluster

* Tested HDFS commands and job submit

* Tested webhdfs through Apache Knox

On Fri, Oct 7, 2016 at 10:35 PM, Karthik Kambatla 
wrote:

> Thanks for putting the RC together, Sangjin.
>
> +1 (binding)
>
> Built from source, deployed pseudo distributed cluster and ran some example
> MR jobs.
>
> On Fri, Oct 7, 2016 at 6:01 PM, Yongjun Zhang  wrote:
>
> > Hi Sangjin,
> >
> > Thanks a lot for your work here.
> >
> > My +1 (binding).
> >
> > - Downloaded both binary and src tarballs
> > - Verified md5 checksum and signature for both
> > - Build from source tarball
> > - Deployed 2 pseudo clusters, one with the released tarball and the other
> > with what I built from source, and did the following on both:
> > - Run basic HDFS operations, and distcp jobs
> > - Run pi job
> > - Examined HDFS webui, YARN webui.
> >
> > Best,
> >
> > --Yongjun
> >
> > > > > * verified basic HDFS operations and Pi job.
> > > > > * Did a sanity check for RM and NM UI.
> >
> >
> > On Fri, Oct 7, 2016 at 5:08 PM, Sangjin Lee  wrote:
> >
> > > I'm casting my vote: +1 (binding)
> > >
> > > Regards,
> > > Sangjin
> > >
> > > On Fri, Oct 7, 2016 at 3:12 PM, Andrew Wang 
> > > wrote:
> > >
> > > > Thanks to Chris and Sangjin for working on this release.
> > > >
> > > > +1 binding
> > > >
> > > > * Verified signatures
> > > > * Built from source tarball
> > > > * Started HDFS and did some basic ops
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > On Fri, Oct 7, 2016 at 2:50 PM, Wangda Tan 
> > wrote:
> > > >
> > > > > Thanks Sangjin for cutting this release!
> > > > >
> > > > > +1 (Binding)
> > > > >
> > > > > - Downloaded binary tar ball and setup a single node cluster.
> > > > > - Submit a few applications and which can successfully run.
> > > > >
> > > > > Thanks,
> > > > > Wangda
> > > > >
> > > > >
> > > > > On Fri, Oct 7, 2016 at 10:33 AM, Zhihai Xu  >
> > > > wrote:
> > > > >
> > > > > > Thanks Sangjin for creating release 2.6.5 RC1.
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > * Downloaded and built from source
> > > > > > * Verified md5 checksums and signature
> > > > > > * Deployed a pseudo cluster
> > > > > > * verified basic HDFS operations and Pi job.
> > > > > > * Did a sanity check for RM and NM UI.
> > > > > >
> > > > > > Thanks
> > > > > > zhihai
> > > > > >
> > > > > > On Fri, Oct 7, 2016 at 8:16 AM, Sangjin Lee 
> > > wrote:
> > > > > >
> > > > > > > Thanks Masatake!
> > > > > > >
> > > > > > > Today's the last day for this vote, and I'd like to ask you to
> > try
> > > > out
> > > > > > the
> > > > > > > RC and vote on this today. So far there has been no binding
> vote.
> > > > > Thanks
> > > > > > > again.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Sangjin
> > > > > > >
> > > > > > > On Fri, Oct 7, 2016 at 6:45 AM, Masatake Iwasaki <
> > > > > > > iwasak...@oss.nttdata.co.jp> wrote:
> > > > > > >
> > > > > > > > +1(non-binding)
> > > > > > > >
> > > > > > > > * verified signature and md5.
> > > > > > > > * built with -Pnative on CentOS6 and OpenJDK7.
> > > > > > > > * built documentation and skimmed the contents.
> > > > > > > > * built rpms by bigtop and ran smoke-tests of hdfs, yarn and
> > > > > mapreduce
> > > > > > on
> > > > > > > > 3-node cluster.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Masatake Iwasaki
> > > > > > > >
> > > > > > > > On 10/3/16 09:12, Sangjin Lee wrote:
> > > > > > > >
> > > > > > > >> Hi folks,
> > > > > > > >>
> > > > > > > >> I have pushed a new release candidate (R1) for the Apache
> > Hadoop
> > > > > 2.6.5
> > > > > > > >> release (the next maintenance release in the 2.6.x release
> > > line).
> > > > > RC1
> > > > > > > >> contains fixes to CHANGES.txt, and is otherwise identical to
> > > RC0.
> > > > > > > >>
> > > > > > > >> Below are the details of this release candidate:
> > > > > > > >>
> > > > > > > >> The RC is available for validation at:
> > > > > > > >> http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/.
> > > > > > > >>
> > > > > > > >> The RC tag in git is release-2.6.5-RC1 and its git commit is
> > > > > > > >> 

Re: [REMINDER] How to set fix versions when committing

[Junping Du]

Hi Andrew and all,
 Thanks for the notice on the change. I still concern this rule change may 
cause some confusion from conflicting against our previous rule - no need to 
set trunk version if it is landing on 2.x branch. As we can see, there are 4 
cases of version setting for JIRA landing on trunk and branch-2 now:
1. JIRA with fixed version set to 2.x only before 3.0.0-alpha1 cut off. 
2. JIRA with fixed version set to 2.x only after 3.0.0-alpha1 cut off.
3. JIRA with fixed version set to 2.x and 3.0.0-alpha1 after 3.0.0-alpha1 cut 
off.
4. JIRA with fixed version set to 2.x and 3.0.0-alpha2 after 3.0.0-alpha1 cut 
off

Case 3 and 4 can be easily distinguished, but case 1 and 2 is against our rule 
change here and hard to differentiate unless we want to mark all previous JIRA 
with fixed version for 2.x only to include 3.0.0-alpha1/3.0.0-alpha2. That's a 
tremendous effort and I doubt this should be our option.
Or, my preference is: we should update our rule here a bit to assume all JIRA 
marked fixed version to 2.x only get landed to 3.0.0-alpha1 but in the 
meanwhile, we monitor all JIRAs come after 3.0.0-alpha1 cut off to include 
3.0.0-alpha2 (or latest trunk version).
Thoughts?


Thanks,

Junping


From: Andrew Wang 
Sent: Tuesday, August 30, 2016 2:57 AM
To: common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: [REMINDER] How to set fix versions when committing

Hi all,

I finished the bulk fix version update and just rebranched
branch-3.0.0-alpha1 off of trunk. So, a reminder that the procedure for
setting fix versions has changed slightly from before.

Everything is fully detailed here, the example in particular should help
clarify things:

https://hadoop.apache.org/versioning.html

The short of it though is that if a JIRA is going into trunk or
branch-3.0.0-alpha1, it should also have a 3.0.0-alpha1 or 3.0.0-alpha2
fixVersion set.

Thanks,
Andrew

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Junping Du]

Thanks Vinod for creating new RC for 2.7.3 release.

+1 (binding) based on following verifications:

- Download src and binary tar ball and verify signature (gpg --verify).

- Build from source Java 1.8.0_31-b13 on Mac native successfully.

- Build from source with Java 1.7.0_79-b15 on Ubuntu VM successfully.

- Deploy a pseudo cluster with running some simple MR jobs, like: sleep, pi, 
teragen/terasort, etc. All finished successfully.

- Check RM/NM web UI and go through some pages like: Scheduler, Nodes, 
Application, etc. Everything works correct.

- Verify basic yarn CLIs, check version (for hadoop and yarn), classpath, 
application list, etc. all seems to work fine.

Thanks,

Junping


From: Vinod Kumar Vavilapalli 
Sent: Thursday, August 18, 2016 3:05 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Cc: Vinod Kumar Vavilapalli
Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2

Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - few more issues with RC1 forced a RC2 [2]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Junping Du]

I think Allen's previous comments are very misleading. 
In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) 
shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, 
daemon's restart, etc.) should get flexible for landing. Otherwise, how could 
52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could 
get landed on branch-2 after 2.2.0 release? Most of them are already released. 

Thanks,

Junping

From: Vinod Kumar Vavilapalli 
Sent: Wednesday, August 17, 2016 9:29 PM
To: Allen Wittenauer
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

I always look at CHANGES.txt entries for incompatible-changes and this JIRA 
obviously wasn’t there.

Anyways, this shouldn’t be in any of branch-2.* as committers there clearly 
mentioned that this is an incompatible change.

I am reverting the patch from branch-2* .

Thanks
+Vinod

> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer  
> wrote:
>
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli  
>> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
>> 
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org 
>>  at 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
>> 
>>
>> The release-notes are inside the tar-balls at location 
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
>>  for 
>> your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. 
>> This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
>> 
>> [2]: 2.7.3 release plan: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>> 
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [Release thread] 2.6.5 release activities

[Junping Du]

I am OK with going forward for 2.6.5 release given some people may not be ready 
for migration to 2.7 and we have done some preparation work already. Thanks 
Chris for pushing the release work forward!
About future releases of 2.6 (after 2.6.5) or any other legacy releases, I 
think it worth more discussions. I disagree with what Allen said below - Java 6 
support should be a solid reason for branch-2.6 release keep going. In this 
community, we are so aggressive to drop Java 7 support in 3.0.x release. Here, 
why we are so conservative to keep releasing new bits to support Java 6? Our 
release effort, although driven by different person, should be consistent 
logically. 
I don't think we have clear rules/polices about EOL of legacy releases before. 
This is a bit off topic here. Will start a new thread later for more discussion.

Thanks,

Junping


From: Chris Trezzo <ctre...@gmail.com>
Sent: Friday, August 12, 2016 12:42 AM
To: Karthik Kambatla
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Release thread] 2.6.5 release activities

Thanks everyone for the discussion! Based on what has been said in this
thread, I will move forward on the preparation efforts (working with
Sangjin and the community) for a 2.6.5 release. If there is a strong
objection to this, please let us know.

I see three initial tasks going forward:

1. Fix the pre-commit build for branch-2.6. I am not entirely sure what is
involved here, but will start looking into it using HADOOP-12800 as a
starting point.

2. Look through the unresolved issues still targeted to 2.6.5 and
resolve/re-target as necessary. There are currently 17 of them:
https://issues.apache.org/jira/issues/?jql=(project%20%
3D%20%22Hadoop%20Common%22%20OR%20project%20%3D%20%22Hadoop%20YARN%22%20OR%
20project%20%3D%20%22Hadoop%20Map%2FReduce%22%20OR%
20project%20%3D%20%22Hadoop%20HDFS%22)%20AND%20(
fixVersion%20%3D%202.6.5%20OR%20%22Target%20Version%2Fs%22%
20%3D%202.6.5)%20AND%20(status%20!%3D%20Resolved%20AND%20status%20!%3D%
20Closed)%20ORDER%20BY%20status%20ASC

3. Look through the backport candidates in the spreadsheet in more detail
and backport/drop as necessary. There are currently 15 of them:
https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC
8hTRUemHvYPPzY/edit?usp=sharing

Finally, I think it would be awesome to continue the release end-of-life
discussion. As we get better and better at release cadence it is going to
be really helpful to have an established EOL policy. It will be less
confusing for contributors and help set clear expectations for end users as
to when they need to start making reasonable migration plans, especially if
they want to stay on a well supported release line. I would be happy to
help with this effort as well. It would be great if we could leverage that
discussion and have EOL information for the 2.6 line when we release 2.6.5.

As always, please let me know if I have missed something.

Thanks!
Chris Trezzo

On Thu, Aug 11, 2016 at 1:03 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> Since there is sufficient interest in 2.6.5, we should probably do it. All
> the reasons Allen outlines make sense.
>
> That said, Junping brings up a very important point that we should think of
> for future releases. For a new user or a user that does not directly
> contribute to the project, more stable releases make it hard to pick from.
>
> As Chris T mentioned, the notion of EOL for our releases seems like a good
> idea. However, to come up with any EOLs, we need to have some sort of
> cadence for the releases. While this is hard for major releases (big bang,
> potentially incompatible features), it should be doable for minor releases.
>
> How do people feel about doing a minor release every 6 months, with
> follow-up maintenance releases every 2 months until the next minor and as
> needed after that? That way, we could EOL a minor release a year after its
> initial release? In the future, we could consider shrinking this window. In
> addition to the EOL, this also makes our releases a little more predictable
> for both users and vendors. Note that 2.6.0 went out almost 2 years ago and
> we haven't had a new minor release in 14 months. I am happy to start
> another DISCUSS thread around this if people think it is useful.
>
> Thanks
> Karthik
>
> On Thu, Aug 11, 2016 at 12:50 PM, Allen Wittenauer <
> a...@effectivemachines.com
> > wrote:
>
> >
> > > On Aug 11, 2016, at 8:10 AM, Junping Du <j...@hortonworks.com> wrote:
> > >
> > > Allen, to be clear, I am not against any branch release effort here.
> > However,
> >
> > "I'm not an X but "
> >
> > > as RM for previous releases 2.6.3 and 2.6.4, I f

Re: [Release thread] 2.6.5 release activities

[Junping Du]

Hi Chris,

  Thanks for your response!

  I think I could miss the thread discussion of "[DISCUSS] 2.6.x line 
releases" for something reason. I checked the discussion - Sean claimed that 
HBase community needs 2.6.5, Zhe said they are using 2.6.x releases and Akira 
said that are over new 50 commits land on branch-2.6 since 2.6.4. Do I miss any 
comments there?

  These comments are more like wishes but not giving more clarifications on 
the needs. I would like to hear more specific reasons to not move to 2.7.x 
releases but prefer to upgrade to 2.6.5. If the only reason is about 
expectation management, I think we should claim 2.6.5 is the last branch-2.6 
release after this release work, otherwise people would expect us to maintain 
this branch forever which is impossible and unnecessary. Thoughts?


Thanks,


Junping



From: Chris Trezzo <ctre...@gmail.com>
Sent: Wednesday, August 10, 2016 9:30 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: [Release thread] 2.6.5 release activities

Thanks Jason and Junping for the comments! I will update the spreadsheet for 
HADOOP-13362 and YARN-4794.

As for continuing 2.6.x releases, please see the discussion in the "[DISCUSS] 
2.6.x line releases" thread. Sean, Akira and Zhe all expressed interest in 
additional 2.6.x releases. I started this thread based off of that interest. I 
understand there is a burden to maintaining a large number of branches. I am 
not sure what the community's end-of-life policy is, but maybe we can issue a 
warning with the 2.6.5 release stating when we will stop maintaining the 
release line. This at least gives users some time to make migration plans to a 
newer version.

On Wed, Aug 10, 2016 at 9:36 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Thanks Chris for bring up this discussion.
Before we going to detail discussion of releasing 2.6.5, I have a quick 
question here: do we think it is necessary to continue to release branch-2.6, 
like 2.6.5, etc after 2.7 is out for more than 1 year. Any reason to not 
suggest users to upgrade to 2.7.3 releases for latest fixes which is in 
releasing now?
My major concern on more release efforts on legacy branches is the same with my 
comments on other release plan before - it seems too many releases trains get 
planned at the same time window (2.6.x, 2.7.x, 2.8, 3.0-alpha, 3.1-beta, etc.). 
Not only user could get confusing on this, but also I suspect we don't have so 
many bandwidth in community to push forward so these releases in high quality 
during the same time window - just like Chris Douglas mentioned in another 
email thread on committer activity and bandwidth. IMO, may be it is better to 
focus on limited number of releases and move them faster?

BTW, I agree with Jason that HADOOP-13362 is not needed for branch-2.6 unless 
we backport container metrics related patches there.


Thanks,

Junping

From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
Sent: Wednesday, August 10, 2016 4:14 PM
To: Chris Trezzo; 
common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>
Subject: Re: [Release thread] 2.6.5 release activities

Thanks for organizing this, Chris!
I don't believe HADOOP-13362 is needed since it's related to ContainerMetrics.  
ContainerMetrics weren't added until 2.7 by YARN-2984.
YARN-4794 looks applicable to 2.6.  The change drops right in except it has 
JDK7-isms (multi-catch clause), so it needs a slight change.

Jason

  From: Chris Trezzo <ctre...@gmail.com<mailto:ctre...@gmail.com>>
 To: "common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>" 
<common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>" 
<mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>>; 
"yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>" 
<yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>>
 Sent: Tuesday, August 9, 2016 7:32 PM
 Subject: [Release thread] 2.6.5 release activities

Based on the sentiment in the "[DISCUSS] 2.6.x line releases" thread, I
have moved forward with some of the initial effort in creating a 2.6.5
release. I am forking this thread so we have a dedicated 2.6.5 release
thread.

I have gone through the git logs and gathered a list of JIRAs that are in
branch-2.7 but 

Re: [Release thread] 2.6.5 release activities

[Junping Du]

Thanks Chris for bring up this discussion. 
Before we going to detail discussion of releasing 2.6.5, I have a quick 
question here: do we think it is necessary to continue to release branch-2.6, 
like 2.6.5, etc after 2.7 is out for more than 1 year. Any reason to not 
suggest users to upgrade to 2.7.3 releases for latest fixes which is in 
releasing now?
My major concern on more release efforts on legacy branches is the same with my 
comments on other release plan before - it seems too many releases trains get 
planned at the same time window (2.6.x, 2.7.x, 2.8, 3.0-alpha, 3.1-beta, etc.). 
Not only user could get confusing on this, but also I suspect we don't have so 
many bandwidth in community to push forward so these releases in high quality 
during the same time window - just like Chris Douglas mentioned in another 
email thread on committer activity and bandwidth. IMO, may be it is better to 
focus on limited number of releases and move them faster?

BTW, I agree with Jason that HADOOP-13362 is not needed for branch-2.6 unless 
we backport container metrics related patches there.


Thanks,

Junping

From: Jason Lowe 
Sent: Wednesday, August 10, 2016 4:14 PM
To: Chris Trezzo; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [Release thread] 2.6.5 release activities

Thanks for organizing this, Chris!
I don't believe HADOOP-13362 is needed since it's related to ContainerMetrics.  
ContainerMetrics weren't added until 2.7 by YARN-2984.
YARN-4794 looks applicable to 2.6.  The change drops right in except it has 
JDK7-isms (multi-catch clause), so it needs a slight change.

Jason

  From: Chris Trezzo 
 To: "common-...@hadoop.apache.org" ; 
hdfs-...@hadoop.apache.org; "mapreduce-dev@hadoop.apache.org" 
; "yarn-...@hadoop.apache.org" 

 Sent: Tuesday, August 9, 2016 7:32 PM
 Subject: [Release thread] 2.6.5 release activities

Based on the sentiment in the "[DISCUSS] 2.6.x line releases" thread, I
have moved forward with some of the initial effort in creating a 2.6.5
release. I am forking this thread so we have a dedicated 2.6.5 release
thread.

I have gone through the git logs and gathered a list of JIRAs that are in
branch-2.7 but are missing from branch-2.6. I limited the diff to issues
with a commit date after 1/26/2016. I did this because 2.6.4 was cut from
branch-2.6 around that date (http://markmail.org/message/xmy7ebs6l3643o5e)
and presumably issues that were committed to branch-2.7 before then were
already looked at as part of 2.6.4.

I have collected these issues in a spreadsheet and have given them an
initial triage on whether they are candidates for a backport to 2.6.5. The
spreadsheet is sorted by the status of the issues with the potential
backport candidates at the top. Here is a link to the spreadsheet:
https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC8hTRUemHvYPPzY/edit?usp=sharing

As of now, I have identified 16 potential backport candidates. Please take
a look at the list and let me know if there are any that you think should
not be on the list, or ones that you think I have missed. This was just an
initial high-level triage, so there could definitely be issues that are
miss-labeled.

As a side note: we still need to look at the pre-commit build for 2.6 and
follow up with an addendum for HADOOP-12800.

Thanks everyone!
Chris Trezzo


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6726) YARN Registry based AM discovery with retry and in-flight task persistent via JHS

[Junping Du (JIRA)]

Junping Du created MAPREDUCE-6726:
-

 Summary: YARN Registry based AM discovery with retry and in-flight 
task persistent via JHS
 Key: MAPREDUCE-6726
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6726
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: applicationmaster
Reporter: Junping Du
Assignee: Srikanth Sampath


Several tasks will be achieved in this JIRA based on the demo patch in 
MAPREDUCE-6608:
1. AM discovery base on YARN register service. Could be replaced by YARN-4758 
later due to scale up issue.
2. Retry logic for TaskUmbilicalProtocol RPC connection
3. In-flight task recover after AM restart via JHS
4. Configuration to control the behavior compatible with previous when not 
enable this feature (by default).
All security related issues and other concerns discussed in MAPREDUCE-6608 will 
be addressed in follow up JIRAs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Increased use of feature branches

[Junping Du]

Comparing with advantages, I believe the disadvantages of shipping any releases 
directly from trunk are more obvious and significant:
- A lot of commits (incompatible, risky, uncompleted feature, etc.) have to 
wait to commit to trunk or put into a separated branch that could delay feature 
development progress as additional vote process get involved even the feature 
is simple and harmless.

- These commits left in separated branches are isolated and get more chance to 
conflict each other, and more bugs could be involved due to conflicts and/or 
less eyes watching/bless on isolated branches.

- More unnecessary arguments/debates will happen on if some commits should land 
on trunk or a separated branch, just like what we have recently.

- Because branches will get increased massively, more community efforts will be 
spent on review & vote for branches merge that means less effort will be spent 
on other commits review given our review bandwidth is quite short so far.

- For small feature with only 1 or 2 commits, that need three +1 from PMCs will 
increase the bar largely for contributors who just start to contribute on 
Hadoop features but no such sufficient support.

Given these concerns, I am open to other options, like: proposed by Vinod or 
Chris, but rather than to release anything directly from trunk.

- This point doesn't necessarily need to be resolved now though, since again 
we're still doing alphas.
No. I think we have to settle down this first. Without a common agreed and 
transparent release process and branches in community, any release (alpha, 
beta) bits is only called a private release but not a official apache hadoop 
release (even alpha).


Thanks,

Junping

From: Karthik Kambatla 
Sent: Friday, June 10, 2016 7:49 AM
To: Andrew Wang
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [DISCUSS] Increased use of feature branches

Thanks for restarting this thread Andrew. I really hope we can get this
across to a VOTE so it is clear.

I see a few advantages shipping from trunk:

   - The lack of need for one additional backport each time.
   - Feature rot in trunk

Instead of creating branch-3, I recommend creating branch-3.x so we can
continue doing 3.x releases off branch-3 even after we move trunk to 4.x (I
said it :))

On Thu, Jun 9, 2016 at 11:12 PM, Andrew Wang 
wrote:

> Hi all,
>
> On a separate thread, a question was raised about 3.x branching and use of
> feature branches going forward.
>
> We discussed this previously on the "Looking to a Hadoop 3 release" thread
> that has spanned the years, with Vinod making this proposal (building on
> ideas from others who also commented in the email thread):
>
>
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201604.mbox/browser
>
> Pasting here for ease:
>
> On an unrelated note, offline I was pitching to a bunch of
> contributors another idea to deal
> with rotting trunk post 3.x: *Make 3.x releases off of trunk directly*.
>
> What this gains us is that
>  - Trunk is always nearly stable or nearly ready for releases
>  - We no longer have some code lying around in some branch (today’s
> trunk) that is not releasable
> because it gets mixed with other undesirable and incompatible changes.
>  - This needs to be coupled with more discipline on individual
> features - medium to to large
> features are always worked upon in branches and get merged into trunk
> (and a nearing release!)
> when they are ready
>  - All incompatible changes go into some sort of a trunk-incompat
> branch and stay there till
> we accumulate enough of those to warrant another major release.
>
> Regarding "trunk-incompat", since we're still in the alpha stage for 3.0.0,
> there's no need for this branch yet. This aspect of Vinod's proposal was
> still under a bit of discussion; Chris Douglas though we should cut a
> branch-3 for the first 3.0.0 beta, which aligns with my original thinking.
> This point doesn't necessarily need to be resolved now though, since again
> we're still doing alphas.
>
> What we should get consensus on is the goal of keeping trunk stable, and
> achieving that by doing more development on feature branches and being
> judicious about merges. My sense from the Hadoop 3 email thread (and the
> more recent one on the async API) is that people are generally in favor of
> this.
>
> We're just about ready to do the first 3.0.0 alpha, so would greatly
> appreciate everyone's timely response in this matter.
>
> Thanks,
> Andrew
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Why there are so many revert operations on trunk?

[Junping Du]

- We need to at the least force a reset of expectations w.r.t how trunk and 
small / medium / incompatible changes there are treated. We should hold off 
making a release off trunk before this gets fully discussed in the community 
and we all reach a consensus.

+1. We should hold off any release work off trunk before we reach a consensus. 
Or more and more developing work/features could be affected just like Larry 
mentioned.


- Reverts (or revert and move to a feature-branch) shouldn’t have been 
unequivocally done without dropping a note / informing everyone / building 
consensus.

Agree. To revert commits from other committers, I think we need to: 1) provide 
technical evidence/reason that is solid as rack, like: break functionality, 
tests, API compatibility, or significantly offend code convention, etc. 2) 
Making consensus with related contributors/committers based on these technical 
reasons/evidences. Unfortunately, I didn't see we ever do either thing in this 
case.


- Freaking out on -1’s and reverts - we as a community need to be less 
stigmatic about -1s / reverts.

+1. As a community, I believe we all prefer to work in a more friendly 
environment. In many cases, -1 without solid reason will frustrate people who 
are doing contributions. I think we should restraint our -1 unless it is really 
necessary.



Thanks,


Junping



From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Monday, June 06, 2016 9:36 PM
To: Andrew Wang
Cc: Junping Du; Aaron T. Myers; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Why there are so many revert operations on trunk?

Folks,

It is truly disappointing how we are escalating situations that can be resolved 
through basic communication.

Things that shouldn’t have happened
- After a few objections were raised, commits should have simply stopped before 
restarting again but only after consensus
- Reverts (or revert and move to a feature-branch) shouldn’t have been 
unequivocally done without dropping a note / informing everyone / building 
consensus. And no, not even a release-manager gets this free pass. Not on 
branch-2, not on trunk, not anywhere.
- Freaking out on -1’s and reverts - we as a community need to be less 
stigmatic about -1s / reverts.

Trunk releases:
This is the other important bit about huge difference of expectations between 
the two sides w.r.t trunk and branching. Till now, we’ve never made releases 
out of trunk. So in-progress features that people deemed to not need a feature 
branch could go into trunk without much trouble. Given that we are now making 
releases off trunk, I can see (a) the RM saying "no, don’t put in-progress 
stuff and (b) the contributors saying “no we don’t want the overhead of a 
branch”. I’ve raised related topics (but only focusing on incompatible changes) 
before - http://markmail.org/message/m6x73t6srlchywsn - but we never decided 
anything.

We need to at the least force a reset of expectations w.r.t how trunk and small 
/ medium / incompatible changes there are treated. We should hold off making a 
release off trunk before this gets fully discussed in the community and we all 
reach a consensus.

* Without a user API, there's no way for people to use it, so not much
advantage to having it in a release

Since the code is separate and probably won't break any existing code, I
won't -1 if you want to include this in a release without a user API, but
again, I question the utility of including code that can't be used.

Clearly, there are two sides to this argument. One side claims the absence of 
user-facing public / stable APIs, and that for all purposes this is dead-code 
for everyone other than the few early adopters who want to experiment with it. 
The other argument is to not put this code before a user API. Again, I’d 
discuss with fellow community members before making what the other side 
perceives as unacceptable moves.

>From 2.8.0 perspective, it shouldn’t have landed there in the first place - I 
>have been pushing for a release for a while with help only from a few members 
>of the community. But if you say that it has no material impact on the user 
>story, having a by-default switched-off feature that *doesn’t* destabilize the 
>core release, I’d be willing to let it pass.

+Vinod

Re: Why there are so many revert operations on trunk?

[Junping Du]

Thanks Aaron for pointing it out. I didn't see any consensus on HDFS-9924 so I 
think we should bring it here with broader audiences for more discussions.

I saw several very bad practices here:

1. committer (no need to say who) revert all commits from trunk without making 
consensus with all related contributors/committers.

2. Someone's comments on feature branch are very misleading... If I didn't 
remember wrong, feature development doesn't have to go through feature branch 
which is just an optional process. This creative process of feature branch and 
branch committer - I believe the intention is trying to accelerate features 
development but not to slow them down.

3. Someone (again, no need to say who) seems to claim himself as RM for trunk. 
I don't think we need any RM for trunk. Even for RM of 3.0.0-alpha, I think we 
need someone else who demonstrates he/she is more responsible, work hardly and 
carefully and open communication with all community. Only through this, the 
success of Hadoop in age of 3.0 are guranteed.


Thanks,


Junping



From: Aaron T. Myers <a...@cloudera.com>
Sent: Monday, June 06, 2016 4:46 PM
To: Junping Du
Cc: Andrew Wang; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Why there are so many revert operations on trunk?

Junping,

All of this is being discussed on HDFS-9924. Suggest you follow the 
conversation there.

--
Aaron T. Myers
Software Engineer, Cloudera

On Mon, Jun 6, 2016 at 7:20 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi Andrew,

 I just noticed you revert 8 commits on trunk last Friday:

HADOOP-13226

HDFS-10430

HDFS-10431

HDFS-10390

HADOOP-13168

HDFS-10390

HADOOP-13168

HDFS-10346

HADOOP-12957

HDFS-10224

   And I didn't see you have any comments on JIRA or email discussion before 
you did this. I don't think we are legally allowed to do this even as 
committer/PMC member. Can you explain what's your intention to do this?

   BTW, thanks Nicolas to revert all these "illegal" revert operations.



Thanks,


Junping

Why there are so many revert operations on trunk?

[Junping Du]

Hi Andrew,

 I just noticed you revert 8 commits on trunk last Friday:

HADOOP-13226

HDFS-10430

HDFS-10431

HDFS-10390

HADOOP-13168

HDFS-10390

HADOOP-13168

HDFS-10346

HADOOP-12957

HDFS-10224

   And I didn't see you have any comments on JIRA or email discussion before 
you did this. I don't think we are legally allowed to do this even as 
committer/PMC member. Can you explain what's your intention to do this?

   BTW, thanks Nicolas to revert all these "illegal" revert operations.



Thanks,


Junping

Re: [DISCUSS] Set minimum version of Hadoop 3 to JDK8 (HADOOP-11858)

[Junping Du]

bq. We'll need to be strict after the switch: all patches to go into branch 2 
will have to go through yetus as branch-2 patches, then cherry picked to trunk, 
or a separate patch on the same JIRA done for the branch-2. Assuming yetus 
still tests the branch-2 stuff on JDK7, that will check version compatibility 
pre-commit.
+1. That's something I want to say also. We should have separate process for 
backport effort between Trunk and branch-2 then. Otherwise, we could involve 
Java 7 related bug to branch-2 after the backport.

Thanks,

Junping

From: Steve Loughran 
Sent: Wednesday, May 11, 2016 5:20 PM
To: Akira AJISAKA
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [DISCUSS] Set minimum version of Hadoop 3 to JDK8 (HADOOP-11858)

>
> On 10 May 2016, at 16:32, Akira AJISAKA  wrote:
>
> Hi developers,
>
> Before cutting 3.0.0-alpha RC, I'd like to drop JDK7 support in trunk.

+1

Given Robert Kanter first filed the patch to do this —why not give him the 
honour. In fact, why not have a webex screen share of the occasion so we can 
all celebrate?



> Given this is a critical change, I'm thinking we should get the consensus 
> first.
>
> One concern I think is, when the minimum version is set to JDK8, we need to 
> configure Jenkins to disable multi JDK test only in trunk.
>

LGTM

We'll need to be strict after the switch: all patches to go into branch 2 will 
have to go through yetus as branch-2 patches, then cherry picked to trunk, or a 
separate patch on the same JIRA done for the branch-2. Assuming yetus still 
tests the branch-2 stuff on JDK7, that will check version compatibility 
pre-commit
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6680) JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)

[Junping Du (JIRA)]

Junping Du created MAPREDUCE-6680:
-

 Summary: JHS UserLogDir scan algorithm sometime could skip 
directory with update in CloudFS (Azure FileSystem, S3, etc.)
 Key: MAPREDUCE-6680
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6680
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Reporter: Junping Du
Assignee: Junping Du


In our cluster based on a Cloud FileSystem, we notice JHS sometimes could skip 
directory with .jhist file in scanning.
The behavior is like:
First round scan, doesn't found .jhist file:
{noformat}
16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a directory 
with 6 files in it.
16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
...
{noformat}

Then, we see "Scan not needed of ..." for the same directory every 3 minutes 
until application failed as timeout.

>From our analysis, we found the root cause is: most of Cloud File System 
>(Azure, S3, etc.) is truncating file/directory modification time to seconds 
>instead of milliseconds - which could due to limit of http protocol (from 
>discussion at: https://forums.aws.amazon.com/thread.jspa?messageID=476615). 

So if the time sequence is happen to be: latest non .jhist file modification on 
directory happens at T1, directory scanning happens at T2, .jhist file added to 
directory at T3. If we have {{T1< T2 < T3}} and T1 is equal to T3 after 
truncating to seconds, this issue could appear.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (MAPREDUCE-5705) mapreduce.task.io.sort.mb hardcoded cap at 2047

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du reopened MAPREDUCE-5705:
---

> mapreduce.task.io.sort.mb hardcoded cap at 2047
> ---
>
> Key: MAPREDUCE-5705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5705
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.2.0
> Environment: Multinode Dell XD720 cluster Centos6 running HDP2
>Reporter: Joseph Niemiec
>
> mapreduce.task.io.sort.mb is hardcoded to not allow values larger then 2047. 
> If you enter a value larger then this the map tasks will always crash at this 
> line -
> https://github.com/apache/hadoop-mapreduce/blob/HDFS-641/src/java/org/apache/hadoop/mapred/MapTask.java?source=cc#L746
> The nodes at dev site have over 380 GB of Ram each, we are not able to make 
> the best use of large mappers (15GB mappers) because of the hardcoded buffer 
> max. Is there a reason this value has been hardcoded? 
> --
> Also validated on my dev VM. Indeed setting io.sort.mb to 2047 works but 2048 
> fails. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 2.7.3 release plan

[Junping Du]

Thanks for bringing up this topic, Sean. 
When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 haven't 
been committed in so that's why we didn't discuss this earlier.
I remember in JIRA discussion, we treated this layout change as a Blocker bug 
that fixing a significant performance regression before but not a normal 
performance improvement. And I believe HDFS community already did their best 
with careful and patient to deliver the fix and other related patches (like 
upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ rounds 
patch review back and forth by senior committers, not to mention the 
outstanding performance test data in HDFS-8791.
I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. 
However, that needs Vinod's final confirmation who serves as RM for branch-2.7. 
In addition, I didn't see any blocker issue to bring it into 2.6.5 now.
Just my 2 cents.

Thanks,

Junping


From: Sean Busbey 
Sent: Thursday, March 31, 2016 2:57 PM
To: hdfs-...@hadoop.apache.org
Cc: Hadoop Common; yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: 2.7.3 release plan

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
 wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



--
busbey

Re: Looking to a Hadoop 3 release

[Junping Du]

Shall we consolidate effort for 2.8.0 and 3.0.0? It doesn't sounds reasonable 
to have two alpha releases to go in parallel. Is EC feature the main motivation 
of releasing hadoop 3 here? If so, I don't understand why this feature cannot 
land on 2.8.x or 2.9.x as an alpha feature. 
If we release 3.0 in a month like plan proposed below, it means we will have 4 
active releases going in parallel - two alpha releases (2.8 and 3.0) and two 
stable releases (2.6.x and 2.7.x). It brings a lot of challenges in issues 
tracking and patch committing, not even mention the tremendous effort of 
release verification and voting.
I would like to propose to wait 2.8 release become stable (may be 2nd release 
in 2.8 branch cause first release is alpha due to discussion in another email 
thread), then we can move to 3.0 as the only alpha release. In the meantime, we 
can bring more significant features (like ATS v2, etc.) to trunk and 
consolidate stable releases in 2.6.x and 2.7.x. I believe that make life 
easier. :)
Thoughts?

Thanks,

Junping 

From: Yongjun Zhang 
Sent: Friday, February 19, 2016 8:05 PM
To: hdfs-...@hadoop.apache.org
Cc: common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release

Thanks Andrew for initiating the effort!

+1 on pushing 3.x with extended alpha cycle, and continuing the more stable
2.x releases.

--Yongjun

On Thu, Feb 18, 2016 at 5:58 PM, Andrew Wang 
wrote:

> Hi Kai,
>
> Sure, I'm open to it. It's a new major release, so we're allowed to make
> these kinds of big changes. The idea behind the extended alpha cycle is
> that downstreams can give us feedback. This way if we do anything too
> radical, we can address it in the next alpha and have downstreams re-test.
>
> Best,
> Andrew
>
> On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai  wrote:
>
> > Thanks Andrew for driving this. Wonder if it's a good chance for
> > HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note
> it's
> > not an incompatible change, but feel better to be done in the major
> release.
> >
> > Regards,
> > Kai
> >
> > -Original Message-
> > From: Andrew Wang [mailto:andrew.w...@cloudera.com]
> > Sent: Friday, February 19, 2016 7:04 AM
> > To: hdfs-...@hadoop.apache.org; Kihwal Lee 
> > Cc: mapreduce-dev@hadoop.apache.org; common-...@hadoop.apache.org;
> > yarn-...@hadoop.apache.org
> > Subject: Re: Looking to a Hadoop 3 release
> >
> > Hi Kihwal,
> >
> > I think there's still value in continuing the 2.x releases. 3.x comes
> with
> > the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
> > be beta or GA for some number of months. In the meanwhile, it'd be good
> to
> > keep putting out regular, stable 2.x releases.
> >
> > Best,
> > Andrew
> >
> >
> > On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee  >
> > wrote:
> >
> > > Moving Hadoop 3 forward sounds fine. If EC is one of the main
> > > motivations, are we getting rid of branch-2.8?
> > >
> > > Kihwal
> > >
> > >   From: Andrew Wang 
> > >  To: "common-...@hadoop.apache.org" 
> > > Cc: "yarn-...@hadoop.apache.org" ; "
> > > mapreduce-dev@hadoop.apache.org" ;
> > > hdfs-dev 
> > >  Sent: Thursday, February 18, 2016 4:35 PM
> > >  Subject: Re: Looking to a Hadoop 3 release
> > >
> > > Hi all,
> > >
> > > Reviving this thread. I've seen renewed interest in a trunk release
> > > since HDFS erasure coding has not yet made it to branch-2. Along with
> > > JDK8, the shell script rewrite, and many other improvements, I think
> > > it's time to revisit Hadoop 3.0 release plans.
> > >
> > > My overall plan is still the same as in my original email: a series of
> > > regular alpha releases leading up to beta and GA. Alpha releases make
> > > it easier for downstreams to integrate with our code, and making them
> > > regular means features can be included when they are ready.
> > >
> > > I know there are some incompatible changes waiting in the wings (i.e.
> > > HDFS-6984 making FileStatus a PB rather than Writable, some of
> > > HADOOP-9991 bumping dependency versions) that would be good to get in.
> > > If you have changes like this, please set the target version to 3.0.0
> > > and mark them "Incompatible". We can use this JIRA query to track:
> > >
> > >
> > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> > > 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> > > 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> > > op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
> > >
> > > There's some release-related stuff that needs to be sorted out
> > > (namely, the new 

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

[Junping Du]

Thanks Yongjun and Allen for the feedback. I agree that option 2 could be a 
safer option if any concern on option 1. Will defer this change to 2.6.5.

Thanks,

Junping

From: Yongjun Zhang <yzh...@cloudera.com>
Sent: Wednesday, February 10, 2016 7:11 PM
To: hdfs-...@hadoop.apache.org
Cc: Hadoop Common; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

Thanks Junping and Allen.

It'd be nice to have HDFS-9629 but I'm ok with option 2, given the fact
that the issue is not critical (and will be addressed in all future
releases), and the concern Allen raised.

Best,

--Yongjun

On Wed, Feb 10, 2016 at 8:37 AM, Allen Wittenauer <a...@altiscale.com> wrote:

>
> > On Feb 9, 2016, at 6:27 PM, Junping Du <j...@hortonworks.com> wrote:
> >
> > Thanks Yongjun for identifying and proposing this change to 2.6.4. I
> think this is the right thing to do and check for following releases. For
> 2.6.4, it seems unnecessary to create another release candidate for this
> issue as we only kicking off a new RC build when last RC has serious
> problem in functionality. The vote progress is quite smoothly so far, so it
> seems unlikely that we will create a new RC. However, I think there are
> still two options here:
> > Option 1:  in final build, adopt change of HDFS-9629 that only updates
> the footer of Web UI to show year 2016.
> > Option 2: skip HDFS-9629 for 2.6.4 and adopt it later for 2.6.5.
> > I prefer Option 1 as this is a very low risky change without affecting
> any functionality, and we allow non-functional changes (like release date,
> etc.) happen on final build after RC passed. I would like to hear the
> voices in community here before acting for the next step. Thoughts?
> >
>
> I’d think having PMC votes apply to what is not actually the final
> artifact is against the ASF rules.
>
>
>

[RESULT] [VOTE] Release Apache Hadoop 2.6.4 RC0

[Junping Du]

I give my binding +1 to conclude the vote for 2.6.4 RC0. With 14 +1s (7 
binding), and no -1s, the vote passes.

Thanks for everyone who tried the release candidate and voted. Also, especially 
thanks to Kihwal Lee, Jason Lowe, Sangjin Lee, Akira AJISAKA and all who help 
in backporting patches for 2.6.4.

I'll push the release bits and send out an announcement for 2.6.4 soon.

Cheers,

Junping

From: Tsuyoshi Ozawa <oz...@apache.org>
Sent: Tuesday, February 09, 2016 11:22 PM
To: mapreduce-dev@hadoop.apache.org
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

+1(binding)

- Verified signatures.
- Ran tests for Apache Tez with the artifacts. All tests passed.
- Ran local cluster and ran some examples on it.

- Tsuyoshi

On Wed, Feb 10, 2016 at 2:18 AM, Sunil Govind <sunil.gov...@gmail.com> wrote:
> +1 (non-binding) with one note,
>
> - Installed tar ball from source and brought up the cluster
> - Ran few MR jobs and those are working fine.
> - UI and REST apis are also looks fine. Ran few rest queries for this test.
> - Tested node label feature and works fine. (one observation during this
> testing)
>
> One note to add here. "-Dmapreduce.job.node-label-expression" support is
> not there 2.6.4. (MAPREDUCE-6304). So we cannot specify label to
> application while submitting.  I think this also can be ported to 2.6
> branch so it will help to specify labels at submission time.
>
> Thanks and Regards
> Sunil
>
> On Wed, Feb 3, 2016 at 12:31 PM Junping Du <j...@hortonworks.com> wrote:
>
>> Hi community folks,
>>I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
>> maintenance release to follow up 2.6.3.) according to email thread of
>> release plan 2.6.4 [1]. Below is details of this release candidate:
>>
>> The RC is available for validation at:
>> *http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
>> <http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/>*
>>
>> The RC tag in git is: release-2.6.4-RC0
>>
>> The maven artifacts are staged via repository.apache.org at:
>> *https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1028/
>> >*
>>
>> You can find my public key at:
>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>>
>> Please try the release and vote. The vote will run for the usual 5 days.
>>
>> Thanks!
>>
>>
>> Cheers,
>>
>> Junping
>>
>>
>> [1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
>>
>>

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

[Junping Du]

Thanks Yongjun for identifying and proposing this change to 2.6.4. I think this 
is the right thing to do and check for following releases. For 2.6.4, it seems 
unnecessary to create another release candidate for this issue as we only 
kicking off a new RC build when last RC has serious problem in functionality. 
The vote progress is quite smoothly so far, so it seems unlikely that we will 
create a new RC. However, I think there are still two options here:
Option 1:  in final build, adopt change of HDFS-9629 that only updates the 
footer of Web UI to show year 2016.
Option 2: skip HDFS-9629 for 2.6.4 and adopt it later for 2.6.5.
I prefer Option 1 as this is a very low risky change without affecting any 
functionality, and we allow non-functional changes (like release date, etc.) 
happen on final build after RC passed. I would like to hear the voices in 
community here before acting for the next step. Thoughts?

Thanks,

Junping

From: Yongjun Zhang <yzh...@cloudera.com>
Sent: Wednesday, February 03, 2016 8:54 AM
To: Hadoop Common
Cc: hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

Hi Junping,

Thanks a lot for leading the effort on 2.6.4.

We were discussing in HDFS-9629, and have agreement on adding an additional
step in the release process to update the year in the webui footer. See

https://issues.apache.org/jira/browse/HDFS-9629?focusedCommentId=15130043=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15130043

I noticed that the current RC for 2.6.4 still has the year of 2014 (the
example steps in the above link is validated on 2.6.4 RC0). So the steps I
proposed in the link could be applied when we create new RCs.

Thanks Akira for +1 on this proposed additional release step. Once we have
an agreement on the details, we will update the wiki page accordingly.

Best,

--Yongjun


On Tue, Feb 2, 2016 at 11:01 PM, Junping Du <j...@hortonworks.com> wrote:

> Hi community folks,
>I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
> maintenance release to follow up 2.6.3.) according to email thread of
> release plan 2.6.4 [1]. Below is details of this release candidate:
>
> The RC is available for validation at:
> *http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
> <http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/>*
>
> The RC tag in git is: release-2.6.4-RC0
>
> The maven artifacts are staged via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
> <https://repository.apache.org/content/repositories/orgapachehadoop-1028/
> >*
>
> You can find my public key at:
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> Please try the release and vote. The vote will run for the usual 5 days.
>
> Thanks!
>
>
> Cheers,
>
> Junping
>
>
> [1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
>
>

[VOTE] Release Apache Hadoop 2.6.4 RC0

[Junping Du]

Hi community folks,
   I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next 
maintenance release to follow up 2.6.3.) according to email thread of release 
plan 2.6.4 [1]. Below is details of this release candidate:

The RC is available for validation at:
*http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
*

The RC tag in git is: release-2.6.4-RC0

The maven artifacts are staged via repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
*

You can find my public key at:
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the usual 5 days.

Thanks!


Cheers,

Junping


[1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Junping Du]

I give my binding +1 on 2.7.2-RC2 with following check:
- All new commits (155) since 2.7.1 are included in commit log and CHANGES.txt 
which equally to JIRA 
  marked as fixed in 2.7.2.

- Signatures and message digests all are looks good, build get successfully 
from the source tar ball with JDK 7.

- Deployment tests passed on a single node, unsecured cluster:
  - Started HDFS/YARN daemons successfully, check daemon logs are in good 
status.
  - Ran several example jobs, like: PI, sleep, teragen/terasort/teravalidate, 
all are finished successfully.
  - Went through the WebUI for NN (50070), DN (50075), RM(8088), NM (8042) to 
make sure the views are working well.

- Verify MR over distributed cache works fine with teragen job.

- Verify NM restart with work preserving works fine with PI job progress 
recovered and finished after NM restarted.

Thanks,

Junping

From: Tsuyoshi Ozawa 
Sent: Wednesday, January 20, 2016 8:15 PM
To: common-...@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

+1 (binding)

- Verified checksum.
- Built from source code.
- Ran some tests and mr examples.

Thanks,
- Tsuyoshi

On Fri, Jan 15, 2016 at 1:57 PM, Vinod Kumar Vavilapalli
 wrote:
> Hi all,
>
> I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.
>
> As discussed before, this is the next maintenance release to follow up 2.7.1.
>
> The RC is available for validation at: 
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/
>
> The RC tag in git is: release-2.7.2-RC2
>
> The maven artifacts are available via repository.apache.org 
>  at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1027 
> 
>
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
>  for 
> your quick perusal.
>
> As you may have noted,
>  - I terminated the RC1 related voting thread after finding out that we 
> didn’t have a bunch of patches that are already in the released 2.6.3 
> version. After a brief discussion, we decided to keep the parallel 2.6.x and 
> 2.7.x releases incremental, see [4] for this discussion.
>  - The RC0 related voting thread got halted due to some critical issues. It 
> took a while again for getting all those blockers out of the way. See the 
> previous voting thread [3] for details.
>  - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite 
> a bit. This release's related discussion threads are linked below: [1] and 
> [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 
> 
> [2]: Planning Apache Hadoop 2.7.2 
> http://markmail.org/message/iktqss2qdeykgpqk 
> 
> [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
> http://markmail.org/message/5txhvr2qdiqglrwc 
> 
> [4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
> http://markmail.org/thread/n7ljbsnquihn3wlw

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Junping Du]

In addition, all new fixes backport from 2.6.3 doesn't listed in 2.7.2 entry of 
CHANGES.txt but keep listed as 2.6.3. I think this is fine no matter we have 
multiple entries to track the same commit or a single entry to track the 
earliest commit. The only thing matter is the commits in release notes (coming 
from JIRA's Fix Versions) are matching with commits that actually land on the 
branch and each commit is tracked in some entry on CHANGES.txt. I did manually 
check that all 155 fixes are existing in commit log and we have entry in 
CHANGES.txt to track each commit on 2.7.2. 
Let's keep focusing on deployment test of 2.7.2. Thanks!

Thanks,

Junping

From: Akira AJISAKA 
Sent: Saturday, January 16, 2016 7:39 PM
To: common-...@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

Hi Xiao,

 >  From a quick comparison between the releasenotes.html and the
CHANGES.txt
 > files in source tarball, the number of total JIRAs is quite different.

In CHANGES.txt, JIRAs fixed in 2.6.1/2.6.2 are not in 2.7.2.
That is why the number of JIRAs are different.

Regards,
Akira

On 1/16/16 09:11, Xiao Chen wrote:
> Thanks Vinod for preparing the release package.
>
>
> +1 (non-binding).
>
> I verified the following:
>
>
> - Successfully downloaded source tarball, verified md5
> - Ran `mvn apache-rat:check` on source tarball, passed
> - Successfully built from source tarball
> - Successfully started a pseudo distributed cluster
> - Ran some basic hdfs operations, then successfully ran a distcp
>
>
> 1 question:
>  From a quick comparison between the releasenotes.html and the CHANGES.txt
> files in source tarball, the number of total JIRAs is quite different.
> Seems like CHANGES.txt misses some of the fixes, because Searching
> 
> 'project
> in (HDFS, HADOOP, MAPREDUCE, YARN) AND resolution = Fixed AND fixVersion =
> 2.7.2' from JIRA agrees with releasenotes.html (count=155).
>
> Here is how I did the check:
>
>
> -  cat ~/Downloads/releasenotes.html |grep "
> https://issues.apache.org/jira/; | awk -F ">" '{print $3}'|awk -F "<"
> '{print $1}' |wc -l
>
>   returns 155
>
>
> - for f in `find . -name CHANGES.txt |grep -v target`; do cat $f |grep
> -B 1 "Release 2.7.1" |grep .*-[0-9] |grep -v "Release 2.7" ; done|wc 
> -l
>
>  returns 103
>
> I'm not sure whether this is a problem or not.
> Best,
> -Xiao
>
> On Thu, Jan 14, 2016 at 8:57 PM, Vinod Kumar Vavilapalli > wrote:
>
>> Hi all,
>>
>> I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.
>>
>> As discussed before, this is the next maintenance release to follow up
>> 2.7.1.
>>
>> The RC is available for validation at:
>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/
>>
>> The RC tag in git is: release-2.7.2-RC2
>>
>> The maven artifacts are available via repository.apache.org <
>> http://repository.apache.org/> at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1027 <
>> https://repository.apache.org/content/repositories/orgapachehadoop-1027>
>>
>> The release-notes are inside the tar-balls at location
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
>> hosted this at
>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html <
>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for
>> your quick perusal.
>>
>> As you may have noted,
>>   - I terminated the RC1 related voting thread after finding out that we
>> didn’t have a bunch of patches that are already in the released 2.6.3
>> version. After a brief discussion, we decided to keep the parallel 2.6.x
>> and 2.7.x releases incremental, see [4] for this discussion.
>>   - The RC0 related voting thread got halted due to some critical issues.
>> It took a while again for getting all those blockers out of the way. See
>> the previous voting thread [3] for details.
>>   - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by
>> quite a bit. This release's related discussion threads are linked below:
>> [1] and [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes <
>> http://markmail.org/message/oozq3gvd4nhzsaes>
>> [2]: Planning Apache Hadoop 2.7.2
>> http://markmail.org/message/iktqss2qdeykgpqk <
>> http://markmail.org/message/iktqss2qdeykgpqk>
>> [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0:
>> http://markmail.org/message/5txhvr2qdiqglrwc <
>> http://markmail.org/message/5txhvr2qdiqglrwc>
>> [4] Retracted [VOTE] Release Apache Hadoop 

Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale branches

[Junping Du]

There are also several stale branches for YARN-2928 that we should consolidate:
  remotes/origin/YARN-2928
  remotes/origin/YARN-2928-new
  remotes/origin/YARN-2928-old
  remotes/origin/YARN-2928-old-2015-11-09
  remotes/origin/YARN-2928-rebase
  remotes/origin/feature-YARN-2928  (the latest one)
I think we should remove all stale ones and rename the latest one 
feature-YARN-2928 to YARN-2928 to follow our practice for branch development. 
Thoughts?


Thanks,

Junping

From: sjl...@gmail.com  on behalf of Sangjin Lee 

Sent: Friday, January 15, 2016 7:17 AM
To: yarn-...@hadoop.apache.org
Cc: Hadoop Common; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
Vinod Kumar Vavilapalli
Subject: Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale branches

Thanks Vinod for the proposal. I deleted my branch (sjlee/hdfs-merge) the
other day.

Sangjin

On Thu, Jan 14, 2016 at 1:26 PM, Vinod Kumar Vavilapalli  wrote:

> Hi all,
>
> As some of you have noticed, we have an update from ASF infra on git
> branching policy: We no longer have a ASF wide mandate on disallowing
> force-pushes on all branches / tags.
>
> Summarizing information from the INFRA email for the sake of clarity in
> the midst of recent confusion
>  - We now can do force pushes, and branch/tag deletion on any branch or
> tag except refs/tags/rel
>  - Any force pushes will be annotated in the commit-email as “[Forced
> Update!]” for the community to watch out for undesired force-pushes
>  - Only tags under refs/tags/rel are protected from force-push for the
> sake of release-provenance: Essentially, the releases that community votes
> on are archived in their entirety with the development history and we
> cannot alter that once a tag is created. As one might expect.
>
> What this means for us
>  - Stale branches: There are a few stale branches that got accumulated.
> — During this branch moratorium, origin/bracnh-2.8 got created (May be
> as part of HDFS-8785, can’t say for sure)
> — A couple of stale branches that helped 2.6.1 release:
> origin/sjlee/hdfs-merge and origin/ajisakaa/common-merge
> — I’ll wait till EOD tomorrow for any yays/nays and delete them
>  - Feature branch updates: Developers can now go rebase and force-push
> their feature branches.
>  - Mainline branches: Mainline branches like trunk, branch-2 have always
> been configured to avoid force-pushes. In general, force-push continues to
> be recommended mainly for feature branches and definitely not on any
> mainline branches from which we make releases.
>  - Release tags:
> — To follow ASF provenance policy, we will now push the final release
> tags under refs/tags/rel. We will first push the RC tags under where they
> reside now (refs/tags) and if the vote passes, the final tag will be
> created under refs/tags/rel.
> — I’ll update our release wiki page
> http://wiki.apache.org/hadoop/HowToRelease <
> http://wiki.apache.org/hadoop/HowToRelease> with the same details once I
> can get 2.7.2 release done using this updated process.
>  - Existing release tags:
> — There is a general recommendation from INFRA team to take all of our
> existing release tags under "tags" and copy them to “rel”.
> — I’ll wait till EOD tomorrow for any yays/nays and copy existing
> releases under refs/tags/rel following general recommendations.
>
> Any comments / thoughts / questions welcome.
>
> Thanks
> +Vinod

Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

[Junping Du]

Thanks Vinod for working hard on this! It is really a huge effort to take care 
of a release with 154 commits.

I did a sanity check for commits on 2.7.2, all looks fine except HDFS-8767 is 
missing from CHANGES.txt due to original commit issue. I already update it, and 
I think we are in good shape now.​


Thanks,


Junping​



From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Wednesday, January 13, 2016 11:15 PM
To: Junping Du
Cc: mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; 
common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

Okay, did a whole bunch of things on 2.7.2.

Fixing CHANGES.txt entry that Junping reported on the voting thread
 - 56e2dcc HADOOP-5323. Trash documentation should describe its directory 
structure and configurations. (Weiwei Yang via ozawa) Moved entry from HDFS 
CHANGES.txt to that of common.

Detected one more issue: YARN-2859 addendum’s got missed in 2.7.2
 - 2e1c617 YARN-2859 addendum: move the entry from 2.6.2 to 2.6.3 in 
hadoop-yarn/CHANGES.txt.
 - d738e2e YARN-2859.addendum: fix the remaining issue from the previous patch

Also removed multiple duplicate CHANGES.txt entries for MAPREDUCE-6549

Backports
 - 91b7f94 YARN-4326. Fix TestDistributedShell timeout as AHS in 
MiniYarnCluster no longer binds to default port 8188. (Meng Ding via wangda)
 - 3c1b25b HADOOP-12413. AccessControlList should avoid calling getGroupNames 
in isUserInList with empty groups. Contributed by Zhihai Xu. (cherry picked 
from commit b2017d9b032af20044fdf60ddbd1575a554ccb79)
 - 17ad3b1 HADOOP-12526. there are duplicate dependency definitions in pom's 
(sjlee)
 - c3ba72c YARN-4241. Fix typo of property name in yarn-default.xml. 
Contributed by Anthony Rojas.
 - 096ac9e MAPREDUCE-6540. TestMRTimelineEventHandling fails (sjlee)
 - 3c0e4de HDFS-8615. Correct HTTP method in WebHDFS document. Contributed by 
Brahma Reddy Battula.
 - ad829ff MAPREDUCE-6377. JHS sorting on state column not working in webUi. 
Contributed by zhihai xu. (cherry picked from commit 
790a861766dcd60212d83f444f2f96d3acf20a94)
 - 10f23dc HDFS-9431. DistributedFileSystem#concat fails if the target path is 
relative. Contributed by Kazuho Fujii.
 - 10a6298 YARN-4344. NMs reconnecting with changed capabilities can lead to 
wrong cluster resource calculations. Contributed by Varun Vasudev (cherry 
picked from commit d36b6e045f317c94e97cb41a163aa974d161a404)
 - 09d3e47 HDFS-9289. Make DataStreamer#block thread safe and verify genStamp 
in commitBlock. Contributed by Chang Li.
 - 89f6716 MAPREDUCE-5883. "Total megabyte-seconds" in job counters is slightly 
misleading. Contributed by Nathan Roberts (cherry picked from commit 
cab3c7c8892ad33a7eb0955b01e99872ab95e192)
 - 4e84a8b YARN-4365. FileSystemNodeLabelStore should check for root dir 
existence on startup. Contributed by Kuhu Shukla (cherry picked from commit 
f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701)
 - 4765a65 MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause 
duplicate records (wilfreds via rkanter) (cherry picked from commit 
7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)
 - 0fc19f6 YARN-4348. ZKRMStateStore.syncInternal shouldn't wait for sync 
completion for avoiding blocking ZK's event thread. (ozawa)
 - 3e192f8 YARN-4434. NodeManager Disk Checker parameter documentation is not 
correct. Contributed by Weiwei Yang.

Ran these tests that got changed in the above patches, before pushing the 
backports:
 - 
TestAccessControlList,DFSTestUtil,TestCommitBlockWithInvalidGenStamp,TestHDFSConcat,TestLineRecordReader,TestMRTimelineEventHandling,TestDistributedShell,TestFileSystemNodeLabelsStore,TestCapacityScheduler

@Junping, mind giving a look at the branch for sanity checks?

Thanks
+Vinod


On Jan 13, 2016, at 11:02 AM, Vinod Kumar Vavilapalli 
<vino...@apache.org<mailto:vino...@apache.org>> wrote:

Thanks for the comments Tsuyoshi / Andrew / Varun / Junping / Akira.

I agree that where possible we should serialize releases and make them 
incremental w.r.t fixes.

Will roll a new RC for 2.7.2 after the backports.

If there are more thoughts on releases, which I’m sure we will all do have, 
let’s fork the thread off from this voting conversation.

Thanks
+Vinod

On Dec 29, 2015, at 6:50 AM, Junping Du 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:

I am +1 with pulling all of these tickets into 2.7.2.
- For “any fix in 2.6.3 to be there in all releases that get out after 2.6.3 
release date”
Shall we conclude this as a general rule - "any fix in 2.x.y to be there in all 
2.b.c releases (while b>=x) that get out after 2.x.y release date"? I am 
generally fine with this, but just feel it sounds to set too strong 
restrictions among branches. Some fixes could be trivial (test case fix, etc.) 
enough to deserve more flexibility.​ I would prefer this rule only applies on 
critical/blocker fixes, 

Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

[Junping Du]

bq.  Is it difficult to backport to 2.7.x if you're already backporting to 
2.6.x? I don't follow why special casing some class of fixes is desirable.
It is not difficult to backport the commits between 2.6.x and 2.7.x. However, 
it do *difficult* to track exactly for hundreds of commits between them. Taking 
HDFS-9470 as an example, the committer totally forget to merge the commit into 
2.7.2 when it is resolved as fixed in 2.7.2. The commit was merged into 2.6.3 
later but get missed on 2.7.2 RC1. If this is not a critical fix, I don't think 
2.7.2 should get a new RC to wait this commit to land on. That's why 
classifying on priority of fixes are important and desirable when we are facing 
this situation.

bq. Also for maintenance releases, aren't all included fixes supposed to be for 
serious bugs? Minor JIRAs can wait for the next minor release. If there are 
strong reasons to include a minor JIRA in a maintenance release, then maybe 
it's not really a minor JIRA.
If a committer commit a major/minor priority patch on a maintenance release, 
what RM should do? Revert it or upgrade the priority to critical even it 
doesn't belong to critical?
I believe only commit critical/blocker patch to maintenance release can only be 
a general guideline for maintenance release, but not a strict rule for all 
committers in practice. RMs should obey this guideline strictly in cherry-pick 
commits but there are more commits get committed by other committers. The 
committer choose the fixed branch not only by priority but also by target 
branch proposed by patch contributor who may only work on that branch release 
for a long time. I think this target/fix branch negotiation mechanism going on 
well and we shouldn't break it.

Thanks,

Junping


From: Andrew Wang <andrew.w...@cloudera.com>
Sent: Friday, January 08, 2016 7:43 PM
To: common-...@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org; Vinod Kumar Vavilapalli; 
yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

I like monotonic releases since it's simple for users to understand. Is it
difficult to backport to 2.7.x if you're already backporting to 2.6.x? I
don't follow why special casing some class of fixes is desirable.

Also for maintenance releases, aren't all included fixes supposed to be for
serious bugs? Minor JIRAs can wait for the next minor release. If there are
strong reasons to include a minor JIRA in a maintenance release, then maybe
it's not really a minor JIRA.

Best,
Andrew

On Fri, Jan 8, 2016 at 8:43 AM, Akira AJISAKA <ajisa...@oss.nttdata.co.jp>
wrote:

> The general rule sounds good to me.
>
> > "any fix in 2.x.y to be there in all 2.b.c releases (while b>=x) that
> get out after 2.x.y release date"
>
> +1
>
> > I would prefer this rule only applies on critical/blocker fixes, but not
> applies on minor/trivial issues.
>
> +1
>
> Thanks,
> Akira
>
>
> On 12/29/15 23:50, Junping Du wrote:
>
>> I am +1 with pulling all of these tickets into 2.7.2.
>>
>> - For “any fix in 2.6.3 to be there in all releases that get out after
>> 2.6.3 release date”
>>
>> Shall we conclude this as a general rule - "any fix in 2.x.y to be there
>> in all 2.b.c releases (while b>=x) that get out after 2.x.y release date"?
>> I am generally fine with this, but just feel it sounds to set too strong
>> restrictions among branches. Some fixes could be trivial (test case fix,
>> etc.) enough to deserve more flexibility.​ I would prefer this rule only
>> applies on critical/blocker fixes, but not applies on minor/trivial issues.
>>
>> Just 2 cents.
>>
>>
>> Thanks,
>>
>>
>> Junping
>>
>>
>> 
>> From: Vinod Kumar Vavilapalli <vino...@apache.org>
>> Sent: Thursday, December 24, 2015 12:47 AM
>> To: Junping Du
>> Cc: mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org;
>> common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
>> Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC1
>>
>> I retract my -1. I think we will need to discuss this a bit more.
>>
>> Beyond those two tickets, there are a bunch more (totaling to 16) that
>> are in 2.6.3 but *not* in 2.7.2. See this:
>> https://issues.apache.org/jira/issues/?jql=key%20in%20%28HADOOP-12526%2CHADOOP-12413%2CHADOOP-11267%2CHADOOP-10668%2CHADOOP-10134%2CYARN-4434%2CYARN-4365%2CYARN-4348%2CYARN-4344%2CYARN-4326%2CYARN-4241%2CYARN-2859%2CMAPREDUCE-6549%2CMAPREDUCE-6540%2CMAPREDUCE-6377%2CMAPREDUCE-5883%2CHDFS-9431%2CHDFS-9289%2CHDFS-8615%29%20and%20fixVersion%20!%3D%202.7.0
>> <
>> https://issues.apache.org/jira/issues/?jql=key%20in%20(HADOOP-12526,HADOOP-12413,HADOOP-

[DISCUSSION] Release Plan for Apache Hadoop 2.6.4

[Junping Du]

Hello folks,
   Hope everyone had a wonderful holiday and good rest in passed weeks. We 
just had a 2.6.3 release on Dec.17 and now it is a good time to look forward to 
the next branch-2.6 maintenance release - 2.6.4 in the beginning of new year. 
As usual, this release will be based on latest release 2.6.3 but include a 
couple of critical/blocker bug fixes that identified in community recently.

Things Done/TODO:

  + Schedule
-- According to previous email discussion thread in community, we are 
prefering fast-moving way (roughly as monthly) of releasing fixes on 
branch-2.6. So I would expect this release could happen around end of Jan. or 
early of Feb. dependens on if RC & VOTE process going on smoothly. Also, it 
should be after release 2.7.2 to get rid of any possible disturb to that 
release.

  + Branch
-- 2.6.4 will be based on branch-2.6, which has been open to 2.6.4 commits 
for a while. I plan to branch out branch-2.6.4 sometime next week according to 
status of target/fixed fixes.
-- Before 2.6.4 branch is cut off, new commits coming as fix to 2.6.4 is 
good to commit/merge to trunk, branch-2 and branch-2.6.

  + Patches
-- In last couple of days, I went through all critial/blocker bug fixes in 
the coming 2.7.2 release and ping authors/committers on JIRA to ask for the 
same effort on branch-2.6.4. I could do more rounds of the same practice in 
case I could miss something helpful. Also, a kindly reminder from original 
author and committer is also appreciated.
-- So far, 2.6.4 already has 18 fixes as list of [1] shows. However, there 
are totally 55 fixes target for 2.6.4 for now [2], so there is still a large 
gap and we need to work harder to resolve these tickets as many as we can and 
push out some non-critical ones to meet our promise of date.

  + Release
-- Still too early to do anything directly related to RC. Will work on 
resolve fixes in list of [2] first, and appreciate any help on this effort.

Any thoughts? Especially on schedule and list of patches that should be 
included?


Cheers,

Junping

[1] Tickets fixed in 2.6.4:
http://s.apache.org/vJB

[2] Tickets targets for 2.6.4:
http://s.apache.org/JpE



From: Junping Du
Sent: Thursday, December 17, 2015 1:36 PM
To: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
junping...@apache.org
Subject: [RESULT] [VOTE] Release Apache Hadoop 2.6.3 RC0

Hi all in community,
 I give my binding +1 to close the vote for 2.6.3 RC0. With 18 +1s (10 
binding), one +0 (non-binding) and no -1s, the vote passes.

Thanks for everyone who tried the release candidate and voted. Thanks Vinod, 
Sangjin and Steve to provide me advices on release process.

I'll push the release bits and send out an announcement for 2.6.3 soon.

Cheers,

Junping

From: Xuan Gong <xg...@hortonworks.com>
Sent: Thursday, December 17, 2015 12:29 AM
To: common-...@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; junping...@apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

+1 (binding),

Build and deploy the cluster from source code.
Ran a few example jobs and passed successfully.

Xuan Gong

> On Dec 16, 2015, at 4:07 PM, Arpit Agarwal <aagar...@hortonworks.com> wrote:
>
> +1 (binding)
>
> - Verified signatures for source and binary distributions
> - Built jars from source with java 1.7.0_79
> - Deployed single-node pseudo-cluster
> - Ran example map reduce jobs
> - Ran hdfs admin commands, verified NN web UI shows expected usages
>
>
>
> On 12/11/15, 4:16 PM, "Junping Du" <j...@hortonworks.com> wrote:
>
>>
>> Hi all developers in hadoop community,
>>  I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the next 
>> maintenance release to follow up 2.6.2.) according to email thread of 
>> release plan 2.6.3 [1]. Sorry for this RC coming a bit late as several 
>> blocker issues were getting committed until yesterday. Below is the details:
>>
>> The RC is available for validation at:
>> *http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
>> <http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/>*
>>
>> The RC tag in git is: release-2.6.3-RC0
>>
>> The maven artifacts are staged via repository.apache.org at:
>> *https://repository.apache.org/content/repositories/orgapachehadoop-1025/?
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1025/>*
>>
>> You can find my public key at:
>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>>
>> Please try the release and vote. The vote will run for the usual 5 days.
>>
>> Thanks and happy weekend!
>>
>>
>> Cheers,
>>
>> Junping
>>
>>
>> [1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y
>>

Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

[Junping Du]

Checked 2.7.2-RC1 tag which match exactly with 2.7.2 branch.
Downloaded the release bit and deploy a single node cluster which works fine 
with running some example jobs.
Built from src and check signatures, all looks good.
Checked release note which has 138 commits (HADOOP:22, HDFS:42, MAPREDUCE:17, 
YARN:57) match exactly with JIRA fixed list in 2.7.2.
However, when I look at our commit log and CHANGES.txt, I found something we 
are missing:
1. HDFS-9470 and YARN-4424 are missing from the 2.7.2 branch and RC1 tag.
2. HADOOP-5323, HDFS-8767 are missing in CHANGE.txt
For 2, I think we can fix in creating the final tag. For 1, I will let release 
manager to decide if HDFS-9470 & YARN-4424 has to go to 2.7.2. If not, I will 
+1 and we can fix release notes later.

Thanks,

Junping

From: Chang Li 
Sent: Tuesday, December 22, 2015 2:35 PM
To: mapreduce-dev@hadoop.apache.org
Cc: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; Vinod Kumar Vavilapalli
Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

+1 (non binding)
Downloaed src and built on single node cluster.
Ran some MR jobs successfully.
Verified signature.

Thanks,
Chang

On Mon, Dec 21, 2015 at 9:34 PM, Tsuyoshi Ozawa  wrote:

> +1
>
> - downloaded src and bin tar balls and verified signatures.
> - built Tez and Spark with 2.7.2 artifacts and JDK7.
> - ran tests of Tez with 2.7.2 artifacts, it passed.
>
> FYI: YARN-4348, reported by Jian, is one of critical issues of 2.7.2
> release.It's better to release 2.7.3 as soon as possible after this
> release.
>
> Thanks,
> - Tsuyoshi
>
> On Tue, Dec 22, 2015 at 4:51 AM, Wangda Tan  wrote:
> > +1 (binding)
> >
> > - Build & deploy single-node Hadoop from source code
> > - Add/Remove node labels to queues/nodes
> > - Run distributed shell commanding using default/specified node labels
> >
> > Thanks,
> > Wangda
> >
> >
> > On Mon, Dec 21, 2015 at 9:58 AM, Masatake Iwasaki <
> > iwasak...@oss.nttdata.co.jp> wrote:
> >
> >> +1(non-binding)
> >>
> >> - verified mds and signature of source and binary tarball
> >> - started 3 node cluster and ran example jobs such as wordcount and
> >> terasort
> >> - built from source tarball with -Pnative on CentOS 7 and OpenJDK 7
> >> - built site documentation and skimmed the contents
> >>
> >> Thanks,
> >> Masatake Iwasaki
> >>
> >>
> >>
> >> On 12/17/15 11:49, Vinod Kumar Vavilapalli wrote:
> >>
> >>> Hi all,
> >>>
> >>> I've created a release candidate RC1 for Apache Hadoop 2.7.2.
> >>>
> >>> As discussed before, this is the next maintenance release to follow up
> >>> 2.7.1.
> >>>
> >>> The RC is available for validation at:
> >>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/ <
> >>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/>
> >>>
> >>> The RC tag in git is: release-2.7.2-RC1
> >>>
> >>> The maven artifacts are available via repository.apache.org <
> >>> http://repository.apache.org/> at
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1026/
> >>> <
> https://repository.apache.org/content/repositories/orgapachehadoop-1026/
> >>> >
> >>>
> >>> The release-notes are inside the tar-balls at location
> >>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> >>> hosted this at
> >>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html
> for
> >>> quick perusal.
> >>>
> >>> As you may have noted,
> >>>   - The RC0 related voting thread got halted due to some critical
> issues.
> >>> It took a while again for getting all those blockers out of the way.
> See
> >>> the previous voting thread [3] for details.
> >>>   - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by
> >>> quite a bit. This release's related discussion threads are linked
> below:
> >>> [1] and [2].
> >>>
> >>> Please try the release and vote; the vote will run for the usual 5
> days.
> >>>
> >>> Thanks,
> >>> Vinod
> >>>
> >>> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes
> <
> >>> http://markmail.org/message/oozq3gvd4nhzsaes>
> >>> [2]: Planning Apache Hadoop 2.7.2
> >>> http://markmail.org/message/iktqss2qdeykgpqk <
> >>> http://markmail.org/message/iktqss2qdeykgpqk>
> >>> [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0:
> >>> http://markmail.org/message/5txhvr2qdiqglrwc
> >>>
> >>>
> >>>
> >>
>

Re: Highlighting communication on releases

[Junping Du]

Thanks Karthik to bring up the topic. Comparing with small enhancement on email 
distribution for release/branch news, I think a more important thing is to make 
our community (contributor & committer) to be more aware on ongoing 
releases/branches.

I can see two improvements could be helpful here:
  - Beside email thread, we can have a central place (like wiki page: 
HowToContribute/HowToCommit) to keep updating the commit instructions/details 
for each releasing/unreleased branches by release manager.
   Following is a quick example:
   /***
   prior to 2.6: closed to commit
   ...
   2.6.4: commit to trunk, branch-2 and branch-2.6
   2.7.2: commit to trunk, branch-2, branch-2.7 and branch-2.7.2 (assume RC
hasn't been out which is not true today.)
   2.8: commit to trunk, branch-2 and branch-2.8
   2.9: commit to trunk and branch-2
   ...
   ***/
   Both contributor and committer can keep an eye on it.

  - Add a tools in post-commit to check if fix version is match with commits
landing on related branches. If not, send notifications to patch contributor, 
committer (and
may be release manager also) to correct it asap.

Thoughts?

Thanks,

Junping


From: Karthik Kambatla 
Sent: Monday, December 21, 2015 4:59 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Highlighting communication on releases

Hi folks

In the last few months, we have been shipping multiple releases -
maintenance and minor - elevating the quality and purpose of our releases.

With the increase in releases and related communication, I wonder if we
need to highlight release-related communication in some way. Otherwise, it
is easy to miss the details in the plethora of emails on dev lists. For
instance, it appears committers might have missed the detail that
branch-2.8 was cut.

A couple of options I see:

   1. Tag release-related emails with [RELEASE] or some other appropriate
   tag.
   2. Separate mailing list for release specific information. May be, this
   list could be used for other project-level topics as well. common-dev@
   has been overloaded for both common-specific and project-level discussions.
   Also, folks wouldn't have to email all -dev@lists.

Would like to hear others' thoughts on the matter.

Karthik