A heads up that I think we're getting close on the blockers for the first
alpha. Looking at my list, I see two I'd like to get in still: YARN-5270
and HADOOP-13316. Will cut a branch and roll the release once those go in;
my test builds have looked good thus far.
My original plan was to do alphas
I am with Vinod on avoiding merging mostly_complete_branches to trunk since
we are not shipping any release off it. If 3.x releases going off of trunk
is going to help with this, I am fine with that approach. We should still
make sure to keep trunk-incompat small and not include large features.
If we're not starting branch-3/trunk, what would distinguish it from
trunk/trunk-incompat? Is it the same mechanism with different labels?
That may be a reasonable strategy when we create branch-3, as a
release branch for beta. Releasing 3.x from trunk will help us figure
out which
Tx for your replies, Andrew.
>> For exit criteria, how about we time box it? My plan was to do monthly
> alphas through the summer, leading up to beta in late August / early Sep.
> At that point we freeze and stabilize for GA in Nov/Dec.
Time-boxing is a reasonable exit-criterion.
> In this
> On Apr 22, 2016, at 6:10 PM, Vinod Kumar Vavilapalli
> wrote:
>
> Nope.
>
> I’m proposing making a new 3.x release (as has been discussed in this thread)
> off today’s trunk (instead of creating a fresh branch-3) and create a new
> trunk-incompt where incompatible
Great comments Vinod, thanks for replying.
Since trunk is a superset of branch-2.8, I think the two efforts are mostly
aligned. The 2.8 blockers are likely also 3.0 blockers. For example, the
create-release and L JIRAs I mentioned are in this camp. The difference
between the two is the
Nope.
I’m proposing making a new 3.x release (as has been discussed in this thread)
off today’s trunk (instead of creating a fresh branch-3) and create a new
trunk-incompt where incompatible changes that we don’t want in 3.x go.
This is mainly to avoid repeating the “we are not releasing 3.x
> On Apr 22, 2016, at 5:38 PM, Vinod Kumar Vavilapalli
> wrote:
>
> On an unrelated note, offline I was pitching to a bunch of contributors
> another idea to deal with rotting trunk post 3.x: *Make 3.x releases off of
> trunk directly*.
>
> What this gains us is that
> -
; re-test.
>>>>>>>
>>>>>>> Best,
>>>>>>> Andrew
>>>>>>>
>>>>>>> On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai <kai.zh...@intel.com>
>>>> wrote:
>>>>>>>
I kind of echo Junping’s comment too.
While 2.8 and 3.0 don’t need to be serialized in theory, in practice I’m
desperately looking for help on 2.8.0. We haven’t been converging on 2.8.0 what
with 50+ blocker / critical patches still unfinished. If postponing 3.x alpha
to after a 2.8.0 alpha
pressure to upgrade for a while, and get Sean's classloader patch in so
> >> it's slightly less visible.
> >> >
> >> > That means 3.0 is going to be an alpha release, not final.
> >> >
> >> > one thing that could be shared is any build.xml automation of th
t; > one thing that could be shared is any build.xml automation of the
>> release process, to at least take away most of the manual steps in the
>> process, to have something more repeatable.
>> >
>> > -steve
>> >
>> >
>> >> Thanks,
&g
ve
> >
> >
> >> Thanks,
> >>
> >> Junping
> >> ____
> >> From: Yongjun Zhang <yzh...@cloudera.com>
> >> Sent: Friday, February 19, 2016 8:05 PM
> >> To: hdfs-dev@hadoop.apache.org
> >> Cc: co
>> From: Yongjun Zhang <yzh...@cloudera.com>
>> Sent: Friday, February 19, 2016 8:05 PM
>> To: hdfs-dev@hadoop.apache.org
>> Cc: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
>> yarn-...@hadoop.apache.org
>> Subject: Re: Looking to
> From: Yongjun Zhang <yzh...@cloudera.com>
> Sent: Friday, February 19, 2016 8:05 PM
> To: hdfs-dev@hadoop.apache.org
> Cc: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: Looking to a
Hi Junping, thanks for the mail, inline:
On Sat, Feb 20, 2016 at 7:34 AM, Junping Du wrote:
> Shall we consolidate effort for 2.8.0 and 3.0.0? It doesn't sounds
> reasonable to have two alpha releases to go in parallel. Is EC feature the
> main motivation of releasing
fs-dev@hadoop.apache.org
Cc: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
yarn-...@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release
Thanks Andrew for initiating the effort!
+1 on pushing 3.x with extended alpha cycle, and continuing the more stable
2.x releases.
--Yongjun
On Th
kih...@yahoo-inc.com>
> > Cc: mapreduce-...@hadoop.apache.org; common-...@hadoop.apache.org;
> > yarn-...@hadoop.apache.org
> > Subject: Re: Looking to a Hadoop 3 release
> >
> > Hi Kihwal,
> >
> > I think there's still value in continuing the 2.x rele
+1 for the plan to start cutting 3.x alpha releases. Thanks for the
initiative Andrew!
On Fri, Feb 19, 2016 at 6:19 AM, Steve Loughran
wrote:
>
> > On 19 Feb 2016, at 11:27, Dmitry Sivachenko wrote:
> >
> >
> >> On 19 Feb 2016, at 01:35, Andrew Wang
> On 19 Feb 2016, at 11:27, Dmitry Sivachenko wrote:
>
>
>> On 19 Feb 2016, at 01:35, Andrew Wang wrote:
>>
>> Hi all,
>>
>> Reviving this thread. I've seen renewed interest in a trunk release since
>> HDFS erasure coding has not yet made it to
t;common-...@hadoop.apache.org>
Cc: "yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; "
mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>;
hdfs-dev <hdfs-dev@hadoop.apache.org>
Sent: Thursday, February 18, 2016 4:35 PM
Subject: R
@hadoop.apache.org; common-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: Looking to a Hadoop 3 release
>
> Hi Kihwal,
>
> I think there's still value in continuing the 2.x releases. 3.x comes with
> the incompatible bump to a JDK8 runtime, and also the fact that
;
> >> From: Andrew Wang <andrew.w...@cloudera.com>
> >> To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>
> >> Cc: "yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; "
> >> mapreduce-...@hadoop.apache.or
t;yarn-...@hadoop.apache.org>; "
>> mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>;
>> hdfs-dev <hdfs-dev@hadoop.apache.org>
>> Sent: Thursday, February 18, 2016 4:35 PM
>> Subject: Re: Looking to a Hadoop 3 release
>>
>
...@cloudera.com]
Sent: Friday, February 19, 2016 7:04 AM
To: hdfs-dev@hadoop.apache.org; Kihwal Lee <kih...@yahoo-inc.com>
Cc: mapreduce-...@hadoop.apache.org; common-...@hadoop.apache.org;
yarn-...@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release
Hi Kihwal,
I think there's still
To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>
> Cc: "yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; "
> mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>;
> hdfs-dev <hdfs-dev@hadoop.apache.or
..@hadoop.apache.org" <yarn-...@hadoop.apache.org>;
"mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>; hdfs-dev
<hdfs-dev@hadoop.apache.org>
Sent: Thursday, February 18, 2016 4:35 PM
Subject: Re: Looking to a Hadoop 3 release
Hi all,
Reviving this thread. I've
Hi all,
Reviving this thread. I've seen renewed interest in a trunk release since
HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
shell script rewrite, and many other improvements, I think it's time to
revisit Hadoop 3.0 release plans.
My overall plan is still the same
On Mar 5, 2015, at 3:21 PM, Siddharth Seth ss...@apache.org wrote:
2) Simplification of configs - potentially separating client side configs
and those used by daemons. This is another source of perpetual confusion
for users.
+ 1 on this.
sanjay
On Mar 6, 2015, at 5:20 PM, Chris Douglas cdoug...@apache.org wrote:
On Fri, Mar 6, 2015 at 4:32 PM, Vinod Kumar Vavilapalli
vino...@hortonworks.com wrote:
I'd encourage everyone to post their wish list on the Roadmap wiki that
*warrants* making incompatible changes forcing us to go 3.x.
-...@hadoop.apache.org;
yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org
Sent: Wednesday, March 4, 2015 12:15 PM
Subject: Re: Looking to a Hadoop 3 release
Let's not dismiss this quite so handily.
Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while
we
could
-dev@hadoop.apache.org
Cc: common-...@hadoop.apache.org common-...@hadoop.apache.org;
mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org;
yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org
Sent: Wednesday, March 4, 2015 12:15 PM
Subject: Re: Looking to a Hadoop 3 release
Yes, these are the kind of enhancements that need to be proposed and discussed
for inclusion!
Thanks,
+Vinod
On Mar 5, 2015, at 3:21 PM, Siddharth Seth ss...@apache.org wrote:
Some features that come to mind immediately would be
1) enhancements to the RPC mechanics - specifically support
: Re: Looking to a Hadoop 3 release
Let's not dismiss this quite so handily.
Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while we
could make classpath isolation opt-in via configuration, what we really
want longer term is to have it on by default (or just always
: Wednesday, March 4, 2015 12:15 PM
Subject: Re: Looking to a Hadoop 3 release
Let's not dismiss this quite so handily.
Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while we
could make classpath isolation opt-in via configuration, what we really
want longer term is to have
Right, but that doesn't really answer the question….
On Mar 5, 2015, at 10:23 PM, Alejandro Abdelnur tuc...@gmail.com wrote:
If classloader isolation is in place, then dependency versions can freely
be upgraded as won't pollute apps space (things get trickier if there is an
ON/OFF switch).
Since these dependency bumps are very disruptive to downstreams, I want to
predicate upgrading our deps on having classpath isolation on. I think
that's what Tucu was getting at.
Best,
Andrew
On Fri, Mar 6, 2015 at 8:01 AM, Allen Wittenauer a...@altiscale.com wrote:
Right, but that doesn't
The 'resistance' is not so much about a new major release, more so about the
content and the roadmap of the release. Other than the two specific features
raised (the need for breaking compat for them is something that I am debating),
I haven't seen a roadmap of branch-3 about any more features
Moving to JDK8 involves a lot of things
(1) Get Hadoop apps to be able to run on JDK8 and chose JDK8 language
features. This is already possible with the decoupling of apps from the
platform.
(2) Get the platform to run on JDK8. This can be done so that we can run
Hadoop on both JDK8 and
IMO, if part of the community wants to take on the responsibility and work
that takes to do a new major release, we should not discourage them from
doing that.
Having multiple major branches active is a standard practice.
This time around we are not replacing the guts as we did from Hadoop 1 to
Sorry, outlook dequoted Alejandros's comments.
Let me try again with his comments in italic and proofreading of mine
On 05/03/2015 13:59, Steve Loughran
ste...@hortonworks.commailto:ste...@hortonworks.com wrote:
On 05/03/2015 13:05, Alejandro Abdelnur
Sent: Wednesday, March 4, 2015 12:15 PM
Subject: Re: Looking to a Hadoop 3 release
Let's not dismiss this quite so handily.
Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while we
could make classpath isolation opt-in via configuration, what we really
want longer term
I think it'll be useful to have a discussion about what else people would
like to see in Hadoop 3.x - especially if the change is potentially
incompatible. Also, what we expect the release schedule to be for major
releases and what triggers them - JVM version, major features, the need for
On 05/03/2015 13:05, Alejandro Abdelnur
tuc...@gmail.commailto:tuc...@gmail.com wrote:
IMO, if part of the community wants to take on the responsibility and work
that takes to do a new major release, we should not discourage them from
doing that.
Having multiple major branches active is a
If classloader isolation is in place, then dependency versions can freely
be upgraded as won't pollute apps space (things get trickier if there is an
ON/OFF switch).
On Thu, Mar 5, 2015 at 9:21 PM, Allen Wittenauer a...@altiscale.com wrote:
Is there going to be a general upgrade of
Thanks all.
There is an open issue HDFS-6962 (ACLs inheritance conflicts with
umaskmode), for which the incompatibility appears to make it not suitable
for 2.x and it's targetted 3.0, please see:
I've taken the liberty of adding a Hadoop 3 section to the Roadmap wiki
page. In addition to the two things I've been pushing, I also looked
through Allen's list (thanks Allen for making this) and picked out the
shell script rewrite and the removal of HFTP as big changes. This would be
the place
Is there going to be a general upgrade of dependencies? I'm thinking of jetty
jackson in particular.
On Mar 5, 2015, at 5:24 PM, Andrew Wang andrew.w...@cloudera.com wrote:
I've taken the liberty of adding a Hadoop 3 section to the Roadmap wiki
page. In addition to the two things I've been
In general +1 on 3.0.0. Its time. If we start now, it might make it out by
2016. If we start now, downstreamers can start aligning themselves to land
versions that suit at about the same time.
While two big items have been called out as possible incompatible changes,
and there is ongoing
, March 03, 2015 2:30 PM
To: common-...@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
yarn-...@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release
I started pitching in more on that JIRA.
To add, I think we can and should strive for doing
-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
yarn-...@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release
In general +1 on 3.0.0. Its time. If we start now, it might make it out by
2016. If we start now, downstreamers can start aligning themselves to land
versions that suit at about the same time
On Wed, Mar 4, 2015 at 10:46 AM, Stack st...@duboce.net wrote:
In general +1 on 3.0.0. Its time. If we start now, it might make it out by
2016. If we start now, downstreamers can start aligning themselves to land
versions that suit at about the same time.
While two big items have been called
One of the questions that keeps popping up is “what exactly is in trunk?”
As some may recall, I had done some experiments creating the change log based
upon JIRA. While the interest level appeared to be approaching zero, I kept
playing with it a bit and eventually also started playing with
Hi Konst, thanks for taking a look. I think I essentially agree with your
points.
On Mon, Mar 2, 2015 at 11:04 PM, Konstantin Shvachko shv.had...@gmail.com
wrote:
Andrew,
Hadoop 3 seems in general like a good idea to me.
1. I did not understand if you propose to release 3.0 instead of 2.7 or
Hi Akira, thanks for responding,
On Tue, Mar 3, 2015 at 4:04 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp
wrote:
Thanks Andrew for bringing this up.
+1 mostly looks fine but I'm thinking it's not now to cut branch-3.
classpath isolation
IMHO, classpath isolation is a good thing to do.
We
From: Akira AJISAKA ajisa...@oss.nttdata.co.jp
Sent: Tuesday, March 03, 2015 12:04 PM
To: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release
Thanks Andrew for bringing
Between:
* removing -finalize
* breaking HDFS browsing
* changing du’s output (in the 2.7 branch)
* changing various names of metrics (either intentionally or otherwise)
* changing the JDK release
… and probably lots of other stuff in branch-2 I
I started pitching in more on that JIRA.
To add, I think we can and should strive for doing this in a compatible manner,
whatever the approach. Marking and calling it incompatible before we see
proposal/patch seems premature to me. Commented the same on JIRA:
To: common-...@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
yarn-...@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release
I started pitching in more on that JIRA.
To add, I think we can and should strive for doing this in a compatible manner
; mapreduce-...@hadoop.apache.org;
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release
Thanks Andrew for bringing this up.
+1 mostly looks fine but I'm thinking it's not now to cut branch-3.
classpath isolation
IMHO, classpath isolation is a good
Thanks Andrew for bringing this up.
+1 mostly looks fine but I'm thinking it's not now to cut branch-3.
classpath isolation
IMHO, classpath isolation is a good thing to do.
We should pay down the technical dept ASAP. I'm willing to help.
I'm thinking we can cut branch-3 and release 3.0 alpha
Hi devs,
It's been a year and a half since 2.x went GA, and I think we're about due
for a 3.x release.
Notably, there are two incompatible changes I'd like to call out, that will
have a tremendous positive impact for our users.
First, classpath isolation being done at HADOOP-11656, which has
+1
On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com
wrote:
Hi devs,
It's been a year and a half since 2.x went GA, and I think we're about due
for a 3.x release.
Notably, there are two incompatible changes I'd like to call out, that will
have a tremendous positive
+1, this sounds like a good plan to me.
Thanks a lot for volunteering to take this on, Andrew.
Best,
Aaron
On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com
wrote:
Hi devs,
It's been a year and a half since 2.x went GA, and I think we're about due
for a 3.x release.
+1
Regards,
Yi Liu
-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com]
Sent: Tuesday, March 03, 2015 7:20 AM
To: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Looking to a Hadoop 3
Thanks Andrew for the proposal.
+1, and I will be happy to help.
--Yongjun
On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com
wrote:
Hi devs,
It's been a year and a half since 2.x went GA, and I think we're about due
for a 3.x release.
Notably, there are two
Andrew
Thanks for bringing up the issue of moving to Java8. Java8 is important
However, I am not seeing a strong motivation for changing the major number.
We can go to Java8 in the 2.series.
The classpath issue for Hadoop-11656 is too minor to force a major number
change (no pun intended).
Second, bumping the source and target JDK version to JDK8 (related to
HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two
months from now). In the past, we've had issues with our dependencies
discontinuing support for old JDKs, so this will future-proof us.
Is moving to
+1. Would love to help.
On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com wrote:
Hi devs,
It's been a year and a half since 2.x went GA, and I think we're about due
for a 3.x release.
Notably, there are two incompatible changes I'd like to call out, that will
have a
+1 Happy to help too
On Mon, Mar 2, 2015 at 3:57 PM, Yongjun Zhang yzh...@cloudera.com wrote:
Thanks Andrew for the proposal.
+1, and I will be happy to help.
--Yongjun
On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com
wrote:
Hi devs,
It's been a year and a
-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Looking to a Hadoop 3 release
Hi devs,
It's been a year and a half since 2.x went GA, and I think we're about due for
a 3.x release.
Notably, there are two incompatible changes I'd like to call
...@cloudera.com
Sent: Monday, March 02, 2015 3:19 PM
To: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Looking to a Hadoop 3 release
Hi devs,
It's been a year and a half since 2.x went GA, and I think we're about due
; mapreduce-...@hadoop.apache.org;
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: RE: Looking to a Hadoop 3 release
JDK8 support is in the consideration, looks like many issues were reported and
resolved already.
https://issues.apache.org/jira/browse/HADOOP-11090
-Original
From: Andrew Wang andrew.w...@cloudera.com
Sent: Monday, March 02, 2015 3:19 PM
To: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Looking to a Hadoop 3 release
Hi devs,
It's been a year and a half
Wang andrew.w...@cloudera.com
Sent: Monday, March 02, 2015 3:19 PM
To: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Looking to a Hadoop 3 release
Hi devs,
It's been a year and a half since 2.x went GA, and I think
Thanks as always for the feedback everyone. Some inline comments to Arun's
email, as his were the most extensive:
Given that we already agreed to put in JDK7 in 2.7, and that the
classpath is a fairly minor irritant given some existing solutions (e.g. a
new default classloader), how do you
Andrew,
Hadoop 3 seems in general like a good idea to me.
1. I did not understand if you propose to release 3.0 instead of 2.7 or in
addition?
I think 2.7 is needed at least as a stabilization step for the 2.x line.
2. If Hadoop 3 and 2.x are meant to exist together, we run a risk to
manifest
77 matches
Mail list logo