Re: [VOTE] Hadoop 3.1.x EOL

[Sangjin Lee]

+1

On Thu, Jun 3, 2021 at 7:35 AM Sean Busbey 
wrote:

> +1
>
> > On Jun 3, 2021, at 1:14 AM, Akira Ajisaka  wrote:
> >
> > Dear Hadoop developers,
> >
> > Given the feedback from the discussion thread [1], I'd like to start
> > an official vote
> > thread for the community to vote and start the 3.1 EOL process.
> >
> > What this entails:
> >
> > (1) an official announcement that no further regular Hadoop 3.1.x
> releases
> > will be made after 3.1.4.
> > (2) resolve JIRAs that specifically target 3.1.5 as won't fix.
> >
> > This vote will run for 7 days and conclude by June 10th, 16:00 JST [2].
> >
> > Committers are eligible to cast binding votes. Non-committers are
> welcomed
> > to cast non-binding votes.
> >
> > Here is my vote, +1
> >
> > [1] https://s.apache.org/w9ilb
> > [2]
> https://www.timeanddate.com/worldclock/fixedtime.html?msg=4=20210610T16=248
> >
> > Regards,
> > Akira
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Change project style guidelines to allow line length 100

[Sangjin Lee]

+1 (binding). It's long overdue IMHO.

On Thu, May 20, 2021 at 2:11 PM Gergely Pollak
 wrote:

> I really like this initiative, thank you!
> +1 for line length increase to 100 characters.
>
> Regards,
>   Gergely Pollak
>
> On Thu, May 20, 2021 at 8:30 PM Vivek Ratnavel 
> wrote:
>
> > +1 (non-binding) to increase line length to 100 characters. This will
> > definitely help clear most of the checkstyle violations.
> >
> > Thank you Sean for starting this thread!
> >
> > On Thu, May 20, 2021 at 7:17 AM Sean Busbey 
> > wrote:
> >
> > > Hi Bhavik!
> > >
> > > What concerns do you have about back porting patches to earlier release
> > > branches?
> > >
> > > If we change our style guidelines then presumably we can do that for
> all
> > > branches, so a backport from e.g. trunk to branch-3.3 won’t fail a
> style
> > > check on the destination branch unless something changed in the
> > backporting.
> > >
> > > If you are referring to patches for clearing up line length violations,
> > my
> > > usual preference is to aim for my changes to be on all active release
> > > lines. So at least in the case of the patches coming from me or being
> > > committed by me, there’d be effort to make sure all branches end up as
> > easy
> > > to backport to as they were prior to the clean up.
> > >
> > >
> > >
> > > > On May 20, 2021, at 2:27 AM, Bhavik Patel 
> > > wrote:
> > > >
> > > > I am just worried about the backporting of the Jira to child branch!!
> > How
> > > > we are planning to handle this?
> > > >
> > > > On Thu, May 20, 2021, 11:09 AM Qi Zhu <821684...@qq.com  > > 821684...@qq.com>> wrote:
> > > >
> > > >> +1 100 is reasonable.
> > > >>
> > > >>
> > > >>
> > > >> ---Original---
> > > >> From: "Xiaoqiao He" > hexiaoq...@apache.org
> > > >
> > > >> Date: Thu, May 20, 2021 13:35 PM
> > > >> To: "Masatake Iwasaki" > > iwasak...@oss.nttdata.co.jp>;
> > > >> Cc: "Akira Ajisaka"mailto:aajis...@apache.org
> > >;"Hadoop
> > > Common"<
> > > >> common-...@hadoop.apache.org  > > >;"Hdfs-dev" > > hdfs-...@hadoop.apache.org>
> > > >> ;"yarn-dev" > > yarn-...@hadoop.apache.org>;"mapreduce-dev"<
> > > >> mapreduce-dev@hadoop.apache.org  > mapreduce-dev@hadoop.apache.org
> > > >;
> > > >> Subject: Re: [DISCUSS] Change project style guidelines to allow line
> > > >> length 100
> > > >>
> > > >>
> > > >> +1 for <= 100 chars long per line length.
> > > >>
> > > >> On Thu, May 20, 2021 at 10:28 AM Masatake Iwasaki <
> > > >> iwasak...@oss.nttdata.co.jp wrote:
> > > >>
> > > >>  I'm +1 too.
> > > >>  I feel 80 characters limit tends to degrade readability by
> > > introducing
> > > >>  useless line breaks.
> > > >> 
> > > >>  
> > > >> 
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
> > > <
> > >
> >
> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
> > > >
> > > >> 
> > > >> <
> > >
> >
> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
> > > <
> > >
> >
> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
> > > >>
> > > >> ;
> > > >>  I have no inconvenience on 100 characters for using Emacs and
> > > >> side-by-side
> > > >>  diff even on 13-inch MBP.
> > > >> 
> > > >>  Masatake Iwasaki
> > > >> 
> > > >>  On 2021/05/20 11:00, Akira Ajisaka wrote:
> > > >>   I'm +1 to allow <= 100 chars.
> > > >>  
> > > >>   FYI: There were some discussions long before:
> > > >>   -
> > > >> 
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
> > > <
> > >
> >
> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
> > > >
> > > >> 
> > > >> <
> > >
> >
> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
> > > <
> > >
> >
> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
> > > >>;
> > > >>  -
> > > >> 
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E
> > > <
> > >
> >
> https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E
> > > >
> > > >> 
> > > >> <
> > >
> >
> https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E
> > > <
> > >
> >

Re: [VOTE] Force "squash and merge" option for PR merge on github UI

[Sangjin Lee]

+1. Sounds good to me.

On Wed, Jul 17, 2019 at 10:20 AM Iñigo Goiri  wrote:

> +1
>
> On Wed, Jul 17, 2019 at 4:17 AM Steve Loughran  >
> wrote:
>
> > +1 for squash and merge, with whoever does the merge adding the full
> commit
> > message for the logs, with JIRA, contributor(s) etc
> >
> > One limit of the github process is that the author of the commit becomes
> > whoever hit the squash button, not whoever did the code, so it loses the
> > credit they are due. This is why I'm doing local merges (With some help
> > from smart-apply-patch). I think I'll have to explore smart-apply-patch
> to
> > see if I can do even more with it
> >
> >
> >
> >
> > On Wed, Jul 17, 2019 at 7:07 AM Elek, Marton  wrote:
> >
> > > Hi,
> > >
> > > Github UI (ui!) helps to merge Pull Requests to the proposed branch.
> > > There are three different ways to do it [1]:
> > >
> > > 1. Keep all the different commits from the PR branch and create one
> > > additional merge commit ("Create a merge commit")
> > >
> > > 2. Squash all the commits and commit the change as one patch ("Squash
> > > and merge")
> > >
> > > 3. Keep all the different commits from the PR branch but rebase, merge
> > > commit will be missing ("Rebase and merge")
> > >
> > >
> > >
> > > As only the option 2 is compatible with the existing development
> > > practices of Hadoop (1 issue = 1 patch = 1 commit), I call for a lazy
> > > consensus vote: If no objections withing 3 days, I will ask INFRA to
> > > disable the options 1 and 3 to make the process less error prone.
> > >
> > > Please let me know, what do you think,
> > >
> > > Thanks a lot
> > > Marton
> > >
> > > ps: Personally I prefer to merge from local as it enables to sign the
> > > commits and do a final build before push. But this is a different
> story,
> > > this proposal is only about removing the options which are obviously
> > > risky...
> > >
> > > ps2: You can always do any kind of merge / commits from CLI, for
> example
> > > to merge a feature branch together with keeping the history.
> > >
> > > [1]:
> > >
> > >
> >
> https://help.github.com/en/articles/merging-a-pull-request#merging-a-pull-request-on-github
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Sangjin Lee]

+1. Thanks for the work, Marton!

On Fri, Aug 31, 2018 at 8:37 AM Vinod Kumar Vavilapalli 
wrote:

> Is there no way to host the new site and the old site concurrently? And
> link back & forth?
>
> +Vinod
>
>
> > On Aug 31, 2018, at 1:07 AM, Elek, Marton  wrote:
> >
> > Bumping this thread at last time.
> >
> > I have the following proposal:
> >
> > 1. I will request a new git repository hadoop-site.git and import the
> new site to there (which has exactly the same content as the existing site).
> >
> > 2. I will ask infra to use the new repository as the source of
> hadoop.apache.org
> >
> > 3. I will sync manually all of the changes in the next two months back
> to the svn site from the git (release announcements, new committers)
> >
> > IN CASE OF ANY PROBLEM we can switch back to the svn without any problem.
> >
> > If no-one objects within three days, I'll assume lazy consensus and
> start with this plan. Please comment if you have objections.
> >
> > Again: it allows immediate fallback at any time as svn repo will be kept
> as is (+ I will keep it up-to-date in the next 2 months)
> >
> > Thanks,
> > Marton
> >
> >
> > On 06/21/2018 09:00 PM, Elek, Marton wrote:
> >> Thank you very much to bump up this thread.
> >> About [2]: (Just for the clarification) the content of the proposed
> website is exactly the same as the old one.
> >> About [1]. I believe that the "mvn site" is perfect for the
> documentation but for website creation there are more simple and powerful
> tools.
> >> Hugo has more simple compared to jekyll. Just one binary, without
> dependencies, works everywhere (mac, linux, windows)
> >> Hugo has much more powerful compared to "mvn site". Easier to
> create/use more modern layout/theme, and easier to handle the content (for
> example new release announcements could be generated as part of the release
> process)
> >> I think it's very low risk to try out a new approach for the site (and
> easy to rollback in case of problems)
> >> Marton
> >> ps: I just updated the patch/preview site with the recent releases:
> >> ***
> >> * http://hadoop.anzix.net *
> >> ***
> >> On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
> >>> Got pinged about this offline.
> >>>
> >>> Thanks for keeping at it, Marton!
> >>>
> >>> I think there are two road-blocks here
> >>>   (1) Is the mechanism using which the website is built good enough -
> mvn-site / hugo etc?
> >>>   (2) Is the new website good enough?
> >>>
> >>> For (1), I just think we need more committer attention and get
> feedback rapidly and get it in.
> >>>
> >>> For (2), how about we do it in a different way in the interest of
> progress?
> >>>   - We create a hadoop.apache.org/new-site/ where this new site goes.
> >>>   - We then modify the existing web-site to say that there is a new
> site/experience that folks can click on a link and navigate to
> >>>   - As this new website matures and gets feedback & fixes, we finally
> pull the plug at a later point of time when we think we are good to go.
> >>>
> >>> Thoughts?
> >>>
> >>> +Vinod
> >>>
>  On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
> 
>  Hi,
> 
>  I would like to bump this thread up.
> 
>  TLDR; There is a proposed version of a new hadoop site which is
> available from here: https://elek.github.io/hadoop-site-proposal/ and
> https://issues.apache.org/jira/browse/HADOOP-14163
> 
>  Please let me know what you think about it.
> 
> 
>  Longer version:
> 
>  This thread started long time ago to use a more modern hadoop site:
> 
>  Goals were:
> 
>  1. To make it easier to manage it (the release entries could be
> created by a script as part of the release process)
>  2. To use a better look-and-feel
>  3. Move it out from svn to git
> 
>  I proposed to:
> 
>  1. Move the existing site to git and generate it with hugo (which is
> a single, standalone binary)
>  2. Move both the rendered and source branches to git.
>  3. (Create a jenkins job to generate the site automatically)
> 
>  NOTE: this is just about forrest based hadoop.apache.org, NOT about
> the documentation which is generated by mvn-site (as before)
> 
> 
>  I got multiple valuable feedback and I improved the proposed site
> according to the comments. Allen had some concerns about the used
> technologies (hugo vs. mvn-site) and I answered all the questions why I
> think mvn-site is the best for documentation and hugo is best for
> generating site.
> 
> 
>  I would like to finish this effort/jira: I would like to start a
> discussion about using this proposed version and approach as a new site of
> Apache Hadoop. Please let me know what you think.
> 
> 
>  Thanks a lot,
>  Marton
> 
>  -
>  To unsubscribe, e-mail: 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Sangjin Lee]

+1 (binding)

- downloaded the binary tarball and the source tarball and checked
signatures
- verified that the source builds cleanly
- verified that the shaded client jars are correct
- checked the basic pseudo-distributed cluster set-up and checked UI and
logs (hdfs and YARN)
- ran some test jobs successfully
- enabled Timeline Service v.2 and tested the writer/reader with test jobs


On Tue, Dec 12, 2017 at 6:14 PM, Jonathan Hung  wrote:

> Thanks Andrew for the huge effort.
>
> +1 (non-binding)
> - Downloaded binary tarball and verified md5
> - Ran RM HA and verified manual failover
> - Verified add/remove/update scheduler configuration API (CLI/REST) works
> for leveldb/zookeeper backend
> - Verified scheduler configuration changes persisted on restart/failover
> - Verified "yarn rmadmin -refreshQueues" works when scheduler configuration
> API disabled, and does not work when scheduler configuration API enabled
>
>
> Jonathan Hung
>
> On Tue, Dec 12, 2017 at 5:44 PM, Junping Du  wrote:
>
> > Thanks Andrew for pushing new RC for 3.0.0. I was out last week, just get
> > chance to validate new RC now.
> >
> > Basically, I found two critical issues with the same rolling upgrade
> > scenario as where HADOOP-15059 get found previously:
> > HDFS-12920, we changed value format for some hdfs configurations that old
> > version MR client doesn't understand when fetching these configurations.
> > Some quick workarounds are to add old value (without time unit) in
> > hdfs-site.xml to override new default values but will generate many
> > annoying warnings. I provided my fix suggestions on the JIRA already for
> > more discussion.
> > The other one is YARN-7646. After we workaround HDFS-12920, will hit the
> > issue that old version MR AppMaster cannot communicate with new version
> of
> > YARN RM - could be related to resource profile changes from YARN side but
> > root cause are still in investigation.
> >
> > The first issue may not belong to a blocker given we can workaround this
> > without code change. I am not sure if we can workaround 2nd issue so far.
> > If not, we may have to fix this or compromise with withdrawing support of
> > rolling upgrade or calling it a stable release.
> >
> >
> > Thanks,
> >
> > Junping
> >
> > 
> > From: Robert Kanter 
> > Sent: Tuesday, December 12, 2017 3:10 PM
> > To: Arun Suresh
> > Cc: Andrew Wang; Lei Xu; Wei-Chiu Chuang; Ajay Kumar; Xiao Chen; Aaron T.
> > Myers; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC1
> >
> > +1 (binding)
> >
> > + Downloaded the binary release
> > + Deployed on a 3 node cluster on CentOS 7.3
> > + Ran some MR jobs, clicked around the UI, etc
> > + Ran some CLI commands (yarn logs, etc)
> >
> > Good job everyone on Hadoop 3!
> >
> >
> > - Robert
> >
> > On Tue, Dec 12, 2017 at 1:56 PM, Arun Suresh  wrote:
> >
> > > +1 (binding)
> > >
> > > - Verified signatures of the source tarball.
> > > - built from source - using the docker build environment.
> > > - set up a pseudo-distributed test cluster.
> > > - ran basic HDFS commands
> > > - ran some basic MR jobs
> > >
> > > Cheers
> > > -Arun
> > >
> > > On Tue, Dec 12, 2017 at 1:52 PM, Andrew Wang  >
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > As a reminder, this vote closes tomorrow at 12:31pm, so please give
> it
> > a
> > > > whack if you have time. There are already enough binding +1s to pass
> > this
> > > > vote, but it'd be great to get additional validation.
> > > >
> > > > Thanks to everyone who's voted thus far!
> > > >
> > > > Best,
> > > > Andrew
> > > >
> > > >
> > > >
> > > > On Tue, Dec 12, 2017 at 11:08 AM, Lei Xu  wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > * Verified src tarball and bin tarball, verified md5 of each.
> > > > > * Build source with -Pdist,native
> > > > > * Started a pseudo cluster
> > > > > * Run ec -listPolicies / -getPolicy / -setPolicy on /  , and run
> hdfs
> > > > > dfs put/get/cat on "/" with XOR-2-1 policy.
> > > > >
> > > > > Thanks Andrew for this great effort!
> > > > >
> > > > > Best,
> > > > >
> > > > >
> > > > > On Tue, Dec 12, 2017 at 9:55 AM, Andrew Wang <
> > andrew.w...@cloudera.com
> > > >
> > > > > wrote:
> > > > > > Hi Wei-Chiu,
> > > > > >
> > > > > > The patchprocess directory is left over from the create-release
> > > > process,
> > > > > > and it looks empty to me. We should still file a create-release
> > JIRA
> > > to
> > > > > fix
> > > > > > this, but I think this is not a blocker. Would you agree?
> > > > > >
> > > > > > Best,
> > > > > > Andrew
> > > > > >
> > > > > > On Tue, Dec 12, 2017 at 9:44 AM, Wei-Chiu Chuang <
> > > weic...@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Andrew, 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Sangjin Lee]

Thanks Andrew. For the record, the commit id would be
c25427ceca461ee979d30edd7a4b0f50718e6533. I mention that for completeness
because of the mutability of tags.

On Mon, Dec 11, 2017 at 10:31 AM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Sorry, forgot to push the tag. It's up there now.
>
> On Sun, Dec 10, 2017 at 8:31 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> I couldn't find the release tag for RC1 either - is it just me or has the
>> release-process changed?
>>
>> +Vinod
>>
>> > On Dec 10, 2017, at 4:31 PM, Sangjin Lee <sj...@apache.org> wrote:
>> >
>> > Hi Andrew,
>> >
>> > Thanks much for your effort! Just to be clear, could you please state
>> the
>> > git commit id of the RC1 we're voting for?
>> >
>> > Sangjin
>> >
>> > On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang <andrew.w...@cloudera.com>
>> > wrote:
>> >
>> >> Hi all,
>> >>
>> >> Let me start, as always, by thanking the efforts of all the
>> contributors
>> >> who contributed to this release, especially those who jumped on the
>> issues
>> >> found in RC0.
>> >>
>> >> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates
>> 302
>> >> fixed JIRAs since the previous 3.0.0-beta1 release.
>> >>
>> >> You can find the artifacts here:
>> >>
>> >> http://home.apache.org/~wang/3.0.0-RC1/
>> >>
>> >> I've done the traditional testing of building from the source tarball
>> and
>> >> running a Pi job on a single node cluster. I also verified that the
>> shaded
>> >> jars are not empty.
>> >>
>> >> Found one issue that create-release (probably due to the mvn deploy
>> change)
>> >> didn't sign the artifacts, but I fixed that by calling mvn one more
>> time.
>> >> Available here:
>> >>
>> >> https://repository.apache.org/content/repositories/orgapache
>> hadoop-1075/
>> >>
>> >> This release will run the standard 5 days, closing on Dec 13th at
>> 12:31pm
>> >> Pacific. My +1 to start.
>> >>
>> >> Best,
>> >> Andrew
>> >>
>>
>>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Sangjin Lee]

Hi Andrew,

Thanks much for your effort! Just to be clear, could you please state the
git commit id of the RC1 we're voting for?

Sangjin

On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang 
wrote:

> Hi all,
>
> Let me start, as always, by thanking the efforts of all the contributors
> who contributed to this release, especially those who jumped on the issues
> found in RC0.
>
> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
> fixed JIRAs since the previous 3.0.0-beta1 release.
>
> You can find the artifacts here:
>
> http://home.apache.org/~wang/3.0.0-RC1/
>
> I've done the traditional testing of building from the source tarball and
> running a Pi job on a single node cluster. I also verified that the shaded
> jars are not empty.
>
> Found one issue that create-release (probably due to the mvn deploy change)
> didn't sign the artifacts, but I fixed that by calling mvn one more time.
> Available here:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1075/
>
> This release will run the standard 5 days, closing on Dec 13th at 12:31pm
> Pacific. My +1 to start.
>
> Best,
> Andrew
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Sangjin Lee]

On Mon, Nov 20, 2017 at 9:46 PM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Thanks for the spot Sangjin. I think this bug introduced in create-release
> by HADOOP-14835. The multi-pass maven build generates these dummy client
> jars during the site build since skipShade is specified.
>
> This might be enough to cancel the RC. Thoughts?
>

IMO yes. This was one of the key features mentioned in the 3.0 release
notes. I appreciate your effort for the release Andrew!


>
> Best,
> Andrew
>
> On Mon, Nov 20, 2017 at 7:51 PM, Sangjin Lee <sj...@apache.org> wrote:
>
>> I checked the client jars that are supposed to contain shaded
>> dependencies, and they don't look quite right:
>>
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar
>> -rw-r--r--  0 andrew andrew44531 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar
>> -rw-r--r--  0 andrew andrew45533 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0.
>> 0.jar
>> -rw-r--r--  0 andrew andrew47015 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar
>>
>> When I look at what's inside those jar, they only seem to include
>> pom-related files with no class files. Am I missing something?
>>
>> When I build from the source with -Pdist, I do get much bigger jars:
>> total 113760
>> -rw-r--r--  1 sangjinlee  120039211  17055399 Nov 20 17:17
>> hadoop-client-api-3.0.0.jar
>> -rw-r--r--  1 sangjinlee  120039211  20451447 Nov 20 17:19
>> hadoop-client-minicluster-3.0.0.jar
>> -rw-r--r--  1 sangjinlee  120039211  20730866 Nov 20 17:18
>> hadoop-client-runtime-3.0.0.jar
>>
>> Sangjin
>>
>> On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee <sj...@apache.org> wrote:
>>
>>>
>>>
>>> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli <
>>> vino...@apache.org> wrote:
>>>
>>>> Thanks for all the push, Andrew!
>>>>
>>>> Looking at the RC. Went through my usual check-list. Here's my summary.
>>>> Will cast my final vote after comparing and validating my findings with
>>>> others.
>>>>
>>>> Verification
>>>>
>>>>  - [Check] Successful recompilation from source tar-ball
>>>>  - [Check] Signature verification
>>>>  - [Check] Generating dist tarballs from source tar-ball
>>>>  - [Check] Testing
>>>> -- Start NN, DN, RM, NM, JHS, Timeline Service
>>>> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
>>>> grep, pi
>>>> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>>>>
>>>> Issues found during testing
>>>>
>>>> Major
>>>>  - The previously supported way of being able to use different
>>>> tar-balls for different sub-modules is completely broken - common and HDFS
>>>> tar.gz are completely empty.
>>>>  - Cannot enable new UI in YARN because it is under a non-default
>>>> compilation flag. It should be on by default.
>>>>  - One decommissioned node in YARN ResourceManager UI always appears to
>>>> start with, even when there are no NodeManagers that are started yet:  Info
>>>> :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
>>>> in the CLI node -list
>>>>
>>>> Minor
>>>>  - resourcemanager-metrics.out is going into current directory instead
>>>> of log directory
>>>>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
>>>> even work. Not just deprecated in favor of timelineserver as was 
>>>> advertised.
>>>>  - Spurious warnings on CLI
>>>> 17/11/20 17:07:34 INFO conf.Configuration:
>>>> resource-types.xml not found
>>>> 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to
>>>> find 'resource-types.xml'.
>>>>
>>>> Side notes
>>>>
>>>>  - When did we stop putting CHANGES files into the source artifacts?
>>>>  - Even after "mvn install"ing once, shading is repeated again and
>>>> again for every new 'mvn install' even though there are no source changes -
>>>> we should see how this can be avoided.
>>>>  - Compatibility notes
>>>>  

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Sangjin Lee]

I checked the client jars that are supposed to contain shaded dependencies,
and they don't look quite right:

$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar
-rw-r--r--  0 andrew andrew44531 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar
$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar
-rw-r--r--  0 andrew andrew45533 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0.0.jar
-rw-r--r--  0 andrew andrew47015 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar

When I look at what's inside those jar, they only seem to include
pom-related files with no class files. Am I missing something?

When I build from the source with -Pdist, I do get much bigger jars:
total 113760
-rw-r--r--  1 sangjinlee  120039211  17055399 Nov 20 17:17
hadoop-client-api-3.0.0.jar
-rw-r--r--  1 sangjinlee  120039211  20451447 Nov 20 17:19
hadoop-client-minicluster-3.0.0.jar
-rw-r--r--  1 sangjinlee  120039211  20730866 Nov 20 17:18
hadoop-client-runtime-3.0.0.jar

Sangjin

On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee <sj...@apache.org> wrote:

>
>
> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> Thanks for all the push, Andrew!
>>
>> Looking at the RC. Went through my usual check-list. Here's my summary.
>> Will cast my final vote after comparing and validating my findings with
>> others.
>>
>> Verification
>>
>>  - [Check] Successful recompilation from source tar-ball
>>  - [Check] Signature verification
>>  - [Check] Generating dist tarballs from source tar-ball
>>  - [Check] Testing
>> -- Start NN, DN, RM, NM, JHS, Timeline Service
>> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
>> grep, pi
>> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>>
>> Issues found during testing
>>
>> Major
>>  - The previously supported way of being able to use different tar-balls
>> for different sub-modules is completely broken - common and HDFS tar.gz are
>> completely empty.
>>  - Cannot enable new UI in YARN because it is under a non-default
>> compilation flag. It should be on by default.
>>  - One decommissioned node in YARN ResourceManager UI always appears to
>> start with, even when there are no NodeManagers that are started yet:  Info
>> :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
>> in the CLI node -list
>>
>> Minor
>>  - resourcemanager-metrics.out is going into current directory instead of
>> log directory
>>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
>> even work. Not just deprecated in favor of timelineserver as was advertised.
>>  - Spurious warnings on CLI
>> 17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml
>> not found
>> 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find
>> 'resource-types.xml'.
>>
>> Side notes
>>
>>  - When did we stop putting CHANGES files into the source artifacts?
>>  - Even after "mvn install"ing once, shading is repeated again and again
>> for every new 'mvn install' even though there are no source changes - we
>> should see how this can be avoided.
>>  - Compatibility notes
>> -- NM's env list is curtailed unlike in 2.x (For e.g,
>> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
>> -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
>> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>>
>
> Sleep has always been in the jobclient test jar as long as I can remember,
> so it's not new for 3.0.
>
>
>>
>> Thanks
>> +Vinod
>>
>> > On Nov 14, 2017, at 1:34 PM, Andrew Wang <andrew.w...@cloudera.com>
>> wrote:
>> >
>> > Hi folks,
>> >
>> > Thanks as always to the many, many contributors who helped with this
>> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>> > available here:
>> >
>> > http://people.apache.org/~wang/3.0.0-RC0/
>> >
>> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>> >
>> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>> > additions include the merge of YARN resource types, API-based
>> configuration
>> > of the CapacityScheduler, and HDFS router-based federation.
>> >
>> > I've done my traditional testing with a pseudo cluster and a Pi job. My
>> +1
>> > to start.
>> >
>> > Best,
>> > Andrew
>>
>>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Sangjin Lee]

On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli  wrote:

> Thanks for all the push, Andrew!
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
> Will cast my final vote after comparing and validating my findings with
> others.
>
> Verification
>
>  - [Check] Successful recompilation from source tar-ball
>  - [Check] Signature verification
>  - [Check] Generating dist tarballs from source tar-ball
>  - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> Issues found during testing
>
> Major
>  - The previously supported way of being able to use different tar-balls
> for different sub-modules is completely broken - common and HDFS tar.gz are
> completely empty.
>  - Cannot enable new UI in YARN because it is under a non-default
> compilation flag. It should be on by default.
>  - One decommissioned node in YARN ResourceManager UI always appears to
> start with, even when there are no NodeManagers that are started yet:  Info
> :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
> in the CLI node -list
>
> Minor
>  - resourcemanager-metrics.out is going into current directory instead of
> log directory
>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even
> work. Not just deprecated in favor of timelineserver as was advertised.
>  - Spurious warnings on CLI
> 17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml
> not found
> 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
>
> Side notes
>
>  - When did we stop putting CHANGES files into the source artifacts?
>  - Even after "mvn install"ing once, shading is repeated again and again
> for every new 'mvn install' even though there are no source changes - we
> should see how this can be avoided.
>  - Compatibility notes
> -- NM's env list is curtailed unlike in 2.x (For e.g,
> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
> -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>

Sleep has always been in the jobclient test jar as long as I can remember,
so it's not new for 3.0.


>
> Thanks
> +Vinod
>
> > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
> wrote:
> >
> > Hi folks,
> >
> > Thanks as always to the many, many contributors who helped with this
> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> > available here:
> >
> > http://people.apache.org/~wang/3.0.0-RC0/
> >
> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> >
> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> > additions include the merge of YARN resource types, API-based
> configuration
> > of the CapacityScheduler, and HDFS router-based federation.
> >
> > I've done my traditional testing with a pseudo cluster and a Pi job. My
> +1
> > to start.
> >
> > Best,
> > Andrew
>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Sangjin Lee]

I recall this discussion about a couple of years ago:
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E

On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <ste...@hortonworks.com>
wrote:

> I'd have assumed it would have gone in as one single patch, rather than a
> full history. I don't see why the trunk needs all the evolutionary history
> of a build.
>
> What should our policy/process be here?
>
> I do currently plan to merge the s3guard in as one single squashed patch;
> just getting HADOOP-14809 sorted first.
>
>
> > On 30 Aug 2017, at 07:09, Vrushali C <vrushalic2...@gmail.com> wrote:
> >
> > I'm adding my +1 (binding) to conclude the vote.
> >
> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > the merge to trunk shortly. Thanks everyone!
> >
> > Regards
> > Vrushali
> >
> >
> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> > varun.saxena.apa...@gmail.com> wrote:
> >
> >> +1 (binding).
> >>
> >> Kudos to all the team members for their great work!
> >>
> >> Being part of the ATSv2 team, I have been involved with either
> development
> >> or review of most of the JIRAs'.
> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> there
> >> is no impact when ATSv2 is turned off.
> >>
> >> Regards,
> >> Varun Saxena.
> >>
> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >> vrushalic2...@gmail.com> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> >>> will
> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>>
> >>> We have previously completed one merge onto trunk [3] and Timeline
> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> Timeline
> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> >>> confident that the state of the feature meets the criteria to be merged
> >>> onto trunk and we'd love folks to get their hands on it in a test
> capacity
> >>> and provide valuable feedback so that we can make it production-ready.
> >>>
> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> >>> usability improvements based on a new architecture. What we would like
> to
> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> complete end-to-end read/write flow with security and read level
> >>> authorization via whitelists. You should be able to start setting it up
> >>> and
> >>> testing it.
> >>>
> >>> At a high level, the following are the key features that have been
> >>> implemented since alpha1:
> >>> - Security via Kerberos Authentication and delegation tokens
> >>> - Read side simple authorization via whitelist
> >>> - Client configurable entity sort ordering
> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> metrics by
> >>> timerange, pagination, sub-app entities
> >>> - Support for storing sub-application entities (entities that exist
> >>> outside
> >>> the scope of an application)
> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> prefixes,
> >>> configurable hbase cluster
> >>> - Flow level aggregations done as dynamic (table level) coprocessors
> >>> - Uses latest stable HBase release 1.2.6
> >>>
> >>> There are a total of 82 subtasks that were completed as part of this
> >>> effort.
> >>>
> >>> We paid close attention to ensure that once disabled Timeline Service
> v.2
> >>> does not impact existing functionality when disabled (by default).
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Regards,
> >>> Vrushali
> >>>
> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>>
> >>
> >>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Sangjin Lee]

+1 (binding)

I've built the current branch, and checked out a few basic areas including
documentation. Also perused the most recent changes that went in.

Thanks much for the great team work! I look forward to seeing it in action.

Regards,
Sangjin

On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <haiboc...@cloudera.com> wrote:

> +1 from my side.
>
> More from the perspective of ensuring there is no impact of ATSv2 when it
> is off (by default), I deployed the latest YARN-5355 bits into a few
> clusters and ran internal Smoke tests. The tests shows no impact when ATSv2
> is off.
>
> Best,
> Haibo
>
> On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <sun...@apache.org> wrote:
>
> > Thank you very much Vrushali, Rohith, Varun and other folks who made this
> > happen. Great work, really appreciate the same!!
> >
> > +1 (binding) from my side:
> >
> > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > # Accessed new YARN UI which shows various flows/flow activity etc. Seems
> > fine.
> > # Based on code, looks like all apis are compatible.
> > # REST api docs looks fine as well, I guess we could improve that a bit
> > more post merge as well.
> > # Adding to additional thoughts which are discussed here, native service
> > also could publish events to atsv2. I think that work is also happened in
> > branch.
> >
> > Looking forward to a much wider adoption of ATSv2 with more projects.
> >
> > Thanks
> > Sunil
> >
> >
> > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > vrushalic2...@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > will
> > > run for 7 days, and will end August 29 11:00 PM PDT.
> > >
> > > We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >
> > > Since then, we have been working on extending the capabilities of
> > Timeline
> > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > confident that the state of the feature meets the criteria to be merged
> > > onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > > and provide valuable feedback so that we can make it production-ready.
> > >
> > > In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> > > usability improvements based on a new architecture. What we would like
> to
> > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > complete end-to-end read/write flow with security and read level
> > > authorization via whitelists. You should be able to start setting it up
> > and
> > > testing it.
> > >
> > > At a high level, the following are the key features that have been
> > > implemented since alpha1:
> > > - Security via Kerberos Authentication and delegation tokens
> > > - Read side simple authorization via whitelist
> > > - Client configurable entity sort ordering
> > > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> > by
> > > timerange, pagination, sub-app entities
> > > - Support for storing sub-application entities (entities that exist
> > outside
> > > the scope of an application)
> > > - Configurable TTLs (time-to-live) for tables, configurable table
> > prefixes,
> > > configurable hbase cluster
> > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > - Uses latest stable HBase release 1.2.6
> > >
> > > There are a total of 82 subtasks that were completed as part of this
> > > effort.
> > >
> > > We paid close attention to ensure that once disabled Timeline Service
> v.2
> > > does not impact existing functionality when disabled (by default).
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >
> > > Regards,
> > > Vrushali
> > >
> > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> 383.html
> > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >
> >
>

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Sangjin Lee]

Thanks Rohith for that update! Sounds good. Let us know when that testing
is complete too.

On Sat, Aug 19, 2017 at 3:34 AM, Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> Hi Sangjin
>
> Thanks for bringing this point.
> We did similar exercise with current YARN-5355 branch today. Tests are
> validated against default configuration and timeline server v.1.5 as well.
>
> All YARN major features such as RM HA/Restart/work-preserving-restart, NM
> restart, scheduling, sample MR/distributes shell jobs run are validated.
> Everything looks fine.
>
> This testing will also be done once YARN-5355 branch code freezed
> completely as well in couple of days.
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote:
>
> > Kudos to Vrushali and the team for getting ready for this large and
> > important feature! I know a huge team effort went into this. I look
> forward
> > to seeing this merged.
> >
> > I'd like to ask one piece of due diligence. Could you please inspect
> > rigorously to ensure that when disabled Timeline Service v.2 does not
> > impact other features in any way? We did a similar exercise when we had
> the
> > first drop, and it would be good to repeat that... Thanks!
> >
> > Sangjin
> >
> > On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <andrew.w...@cloudera.com>
> > wrote:
> >
> > > Great, thanks Vrushali! Sounds good to me.
> > >
> > > I have a few procedural release notes comments I'll put on YARN-5355,
> to
> > > make sure we advertise this to our users appropriately.
> > >
> > > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > > vrushal...@gmail.com> wrote:
> > >
> > > > Hi Andrew,
> > > >
> > > > Thanks for your response!
> > > >
> > > > There have been no changes to existing APIs since alpha1.
> > > >
> > > > We at Twitter have tested the feature to demonstrate it works at what
> > we
> > > > consider moderate scale but this did not include the security related
> > > > testing. The security testing is in progress at present by Timeline
> > > Service
> > > > V2 team in the community and we think we will have more details on
> this
> > > > very soon.
> > > >
> > > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what
> we
> > > > think of as "merge-blockers". The issues being targeted for merge are
> > in
> > > > [link1] below. There are about 59 jiras of which 56 are completed.
> > > >
> > > > We plan to make a new umbrella jira after the merge to trunk. We will
> > > then
> > > > create a new branch with the new jira name and move these open jiras
> > > under
> > > > YARN-5355 as subtasks of that new umbrella jira.
> > > >
> > > > thanks
> > > > Vrushali
> > > > [link1] https://issues.apache.org/jira/projects/YARN/versions/
> 12337991
> > > >
> > > >
> > > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> > andrew.w...@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi Vrushali,
> > > >>
> > > >> Glad to hear this major dev milestone is nearing completion!
> > > >>
> > > >> Repeating my request on other merge [DISCUSS] threads, could you
> > comment
> > > >> on testing and API stability of this merge? Our timeline for beta1
> is
> > > about
> > > >> a month out, so there's not much time to fix things beforehand.
> > > >>
> > > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > > most
> > > >> of these be moved out to a new umbrella? I'm wondering what needs to
> > be
> > > >> completed before sending the merge vote.
> > > >>
> > > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> > the
> > > >> beta1 release date for this feature than others. Hopefully that
> won't
> > be
> > > >> necessary though :)
> > > >>
> > > >> Best,
> > > >> Andrew
> > > >>
> > > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > > >> vrushalic2...@gmail.com> wrote:
> > > >>
> > > >>> Looks like some of the hype

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Sangjin Lee]

gt; omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> >>> 2C%20created%20ASC>
> >>> ]
> >>> - HBase specific improvements [atsv2-hbase
> >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - REST API additions and improvements [timeline-reader
> >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - Reader side simple authorization via whitelist [YARN-6820]
> >>>
> >>> We would love to get your thoughts on this before we open a real voting
> >>> thread.
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Thanks
> >>> Vrushali
> >>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache
> .
> >>> org/msg23897.html
> >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
> >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> >>> ache/hadoop/commits/YARN-5355
> >>>
> >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> >>> vrushal...@gmail.com> wrote:
> >>>
> >>> > Hi All
> >>> >
> >>> > I’d like to open a discussion for merging Timeline Service v.2
> >>> (YARN-5355)
> >>> > to trunk in a few weeks.
> >>> >
> >>> > We have previously completed one merge onto trunk [1] and Timeline
> >>> Service
> >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >
> >>> > Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> > Service v2 in a feature branch [2]. There are a few related issues
> >>> pending
> >>> > that are being actively worked upon & tested. As soon as they are
> >>> resolved,
> >>> > we plan on starting a merge vote within the next 2 weeks. The goal is
> >>> to
> >>> > get this in for Hadoop3 beta.
> >>> >
> >>> > We have paid close attention to ensure that once disabled Timeline
> >>> Service
> >>> > v.2 does not impact existing functionality when disabled (by
> default).
> >>> >
> >>> > At a high level, following are the key features that have been
> >>> implemented
> >>> > since 3.0.0-alpha1:
> >>> > - Security (via Kerberos Authentication & delegation tokens) at the
> >>> writer
> >>> > [YARN-3053]
> >>> > - Timeline server usability improvements [timeline-server
> >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> > ]
> >>> > - HBase specific improvements [atsv2-hbase
> >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> >>> ed%20DESC%2C%
> >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> > ]
> >>> > - REST API additions & improvements [timeline-reader
> >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> >>> ty%20DESC%2C%
> >>> > 20created%20ASC>
> >>> > ]
> >>> > - Reader side simple authorization via whitelist [YARN-6820]
> >>> >
> >>> > We would love to get your thoughts on these and more before we open a
> >>> real
> >>> > voting thread.
> >>> >
> >>> > Special thanks to a team of folks who worked hard and contributed
> >>> towards
> >>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
> >>> Varun
> >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>> >
> >>> > Thanks!
> >>> > Vrushali
> >>> >
> >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> >>> > apache.org/msg23897.html
> >>> > [2] feature branch YARN-5355 commits: https://github.com/
> >>> > apache/hadoop/commits/YARN-5355
> >>> >
> >>>
> >>
> >>
> >
>

Re: About 2.7.4 Release

[Sangjin Lee]

I don't think there should be any linkage between releasing 2.8.0 and
2.7.4. If we have a volunteer for releasing 2.7.4, we should go full speed
ahead. We still need a volunteer from a PMC member or a committer as some
tasks may require certain privileges, but I don't think it precludes
working with others to close down the release.

I for one would like to see more frequent releases, and being able to
automate release steps more would go a long way.

On Tue, Mar 7, 2017 at 2:16 AM, Marton Elek  wrote:

> Is there any reason to wait for 2.8 with 2.7.4?
>
> Unfortunately the previous  thread about release cadence has been ended
> without final decision. But if I understood well, there was more or less an
> agreement about that it would be great to achieve more frequent releases,
> if possible (with or without written rules and EOL policy).
>
> I personally prefer to be more closer to the scheduling part of the
> proposal:
>
> "A minor release on the latest major line should be every 6 months, and a
> maintenance release on a minor release (as there may be concurrently
> maintained minor releases) every 2 months".
>
> I don't know what is the hardest part of creating new minor/maintenance
> releases. But if the problems are technical (smoketesting, unit tests, old
> release script, anything else) I would be happy to do any task for new
> maintenance releases (or more frequent releases).
>
> Regards,
> Marton
>
>
> 
> From: Akira Ajisaka 
> Sent: Tuesday, March 07, 2017 7:34 AM
> To: Brahma Reddy Battula; Hadoop Common; yarn-...@hadoop.apache.org;
> Hdfs-dev; mapreduce-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> Probably 2.8.0 will be released soon.
> https://issues.apache.org/jira/browse/HADOOP-13866?
> focusedCommentId=15898379=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15898379
>
> I'm thinking 2.7.4 release process starts after 2.8.0 release,
> so 2.7.4 will be released in April or May. (hopefully)
>
> Thoughts?
>
> Regards,
> Akira
>
> On 2017/03/01 21:01, Brahma Reddy Battula wrote:
> > Hi All
> >
> > It has been six months for branch-2.7 release.. is there any near plan
> for 2.7.4..?
> >
> >
> > Thanks
> > Brahma Reddy Battula
> >
> >
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Sangjin Lee]

File MAPREDUCE-6836 <https://issues.apache.org/jira/browse/MAPREDUCE-6836>.

On Wed, Jan 25, 2017 at 2:35 PM, Sangjin Lee <sj...@apache.org> wrote:

> Sorry for missing the cutoff.
>
> I'm also +1 (binding) if that counts.
>
> I did find a small issue with MR, and I'll file a separate JIRA, but it
> shouldn't stop the release.
>
> On Wed, Jan 25, 2017 at 12:06 PM, Andrew Wang <andrew.w...@cloudera.com>
> wrote:
>
>> Thanks guys. With that, I'm going to close the VOTE for 3.0.0-alpha2 with
>> the following vote totals:
>>
>> 5 binding +1s (Arun got his vote in just in time ;)
>> 7 non-binding +1s
>> 0 -1's
>>
>> Thanks to everyone who tried out the RC! I'll get started on publishing
>> the
>> release.
>>
>> Best,
>> Andrew
>>
>> On Wed, Jan 25, 2017 at 12:05 PM, Arun Suresh <arun.sur...@gmail.com>
>> wrote:
>>
>> > Thanks for all the work on this Andrew. I agree with Chris and Karthik.
>> >
>> > +1 (binding) from me as well
>> >   - Ran a 9 node cluster HDFS and YARN cluster
>> >   - Played with some hdfs commands
>> >   - Ran some sleep jobs using YARN ReservationSytstem + Opportunistic
>> > Containers
>> >   - Compiled REEF and Spark against the src.
>> >
>> > Cheers
>> > -Arun
>> >
>> >
>> > On Wed, Jan 25, 2017 at 12:01 PM, Karthik Kambatla <ka...@cloudera.com>
>> > wrote:
>> >
>> >> I feel the same way as Chris.
>> >>
>> >> +1 on the current RC. And, open to changes to RN and License.
>> >>
>> >> On Wed, Jan 25, 2017 at 11:59 AM, Chris Douglas <
>> chris.doug...@gmail.com>
>> >> wrote:
>> >>
>> >> > On Wed, Jan 25, 2017 at 11:42 AM, Andrew Wang <
>> andrew.w...@cloudera.com
>> >> >
>> >> > wrote:
>> >> > > Chris and Karthik, could you clarify the contingency of your
>> votes? Is
>> >> > > fixing just the release notes sufficient?
>> >> >
>> >> > My +1 was not contingent on any changes.
>> >> >
>> >> > The release is fine as-is. Fixing any subset of the release notes,
>> >> > minicluster jar, and the organization of LICENSE files within jars
>> >> > should not reset the clock on the VOTE. -C
>> >> >
>> >> > > On Wed, Jan 25, 2017 at 11:14 AM, Karthik Kambatla <
>> >> ka...@cloudera.com>
>> >> > > wrote:
>> >> > >
>> >> > >> Thanks for driving the alphas, Andrew. I don't see the need to
>> >> restart
>> >> > the
>> >> > >> vote and I feel it is okay to fix the minor issues before
>> releasing.
>> >> > >>
>> >> > >> +1 (binding). Downloaded source, stood up a pseudo-distributed
>> >> cluster
>> >> > >> with FairScheduler, ran example jobs, and played around with the
>> UI.
>> >> > >>
>> >> > >> Thanks
>> >> > >> Karthik
>> >> > >>
>> >> > >>
>> >> > >> On Fri, Jan 20, 2017 at 2:36 PM, Andrew Wang <
>> >> andrew.w...@cloudera.com>
>> >> > >> wrote:
>> >> > >>
>> >> > >>> Hi all,
>> >> > >>>
>> >> > >>> With heartfelt thanks to many contributors, the RC0 for
>> >> 3.0.0-alpha2 is
>> >> > >>> ready.
>> >> > >>>
>> >> > >>> 3.0.0-alpha2 is the second alpha in the planned 3.0.0 release
>> line
>> >> > leading
>> >> > >>> up to a 3.0.0 GA. It comprises 857 fixes, improvements, and new
>> >> > features
>> >> > >>> since alpha1 was released on September 3rd, 2016.
>> >> > >>>
>> >> > >>> More information about the 3.0.0 release plan can be found here:
>> >> > >>>
>> >> > >>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >> > Hadoop+3.0.0+release
>> >> > >>>
>> >> > >>> The artifacts can be found here:
>> >> > >>>
>> >> > >>> http://home.apache.org/~wang/3.0.0-alpha2-RC0/
>> >> > >>>
>> >> > >>> This vote will run 5 days, ending on 01/25/2017 at 2PM pacific.
>> >> > >>>
>> >> > >>> I ran basic validation with a local pseudo cluster and a Pi job.
>> RAT
>> >> > >>> output
>> >> > >>> was clean.
>> >> > >>>
>> >> > >>> My +1 to start.
>> >> > >>>
>> >> > >>> Thanks,
>> >> > >>> Andrew
>> >> > >>>
>> >> > >>
>> >> > >>
>> >> >
>> >>
>> >
>> >
>>
>
>

[jira] [Created] (MAPREDUCE-6836) exception thrown when accessing the job configuration web UI

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6836:
--

 Summary: exception thrown when accessing the job configuration web 
UI
 Key: MAPREDUCE-6836
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6836
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: webapps
Affects Versions: 3.0.0-alpha2
Reporter: Sangjin Lee
Priority: Minor


When I navigate the MR job web UI and click the configuration link, the AM 
shows an exception:
{noformat}
2017-01-25 11:40:55,521 ERROR [qtp2126664214-26] 
org.apache.hadoop.yarn.webapp.Dispatcher: error handling URI: /mapreduc
e/conf/job_1485372765455_0002
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:162)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at 
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at 
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at 
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at 
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at 
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at 
com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at 
org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at 
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:179)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at 
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1458)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:524)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Sangjin Lee]

Sorry for missing the cutoff.

I'm also +1 (binding) if that counts.

I did find a small issue with MR, and I'll file a separate JIRA, but it
shouldn't stop the release.

On Wed, Jan 25, 2017 at 12:06 PM, Andrew Wang 
wrote:

> Thanks guys. With that, I'm going to close the VOTE for 3.0.0-alpha2 with
> the following vote totals:
>
> 5 binding +1s (Arun got his vote in just in time ;)
> 7 non-binding +1s
> 0 -1's
>
> Thanks to everyone who tried out the RC! I'll get started on publishing the
> release.
>
> Best,
> Andrew
>
> On Wed, Jan 25, 2017 at 12:05 PM, Arun Suresh 
> wrote:
>
> > Thanks for all the work on this Andrew. I agree with Chris and Karthik.
> >
> > +1 (binding) from me as well
> >   - Ran a 9 node cluster HDFS and YARN cluster
> >   - Played with some hdfs commands
> >   - Ran some sleep jobs using YARN ReservationSytstem + Opportunistic
> > Containers
> >   - Compiled REEF and Spark against the src.
> >
> > Cheers
> > -Arun
> >
> >
> > On Wed, Jan 25, 2017 at 12:01 PM, Karthik Kambatla 
> > wrote:
> >
> >> I feel the same way as Chris.
> >>
> >> +1 on the current RC. And, open to changes to RN and License.
> >>
> >> On Wed, Jan 25, 2017 at 11:59 AM, Chris Douglas <
> chris.doug...@gmail.com>
> >> wrote:
> >>
> >> > On Wed, Jan 25, 2017 at 11:42 AM, Andrew Wang <
> andrew.w...@cloudera.com
> >> >
> >> > wrote:
> >> > > Chris and Karthik, could you clarify the contingency of your votes?
> Is
> >> > > fixing just the release notes sufficient?
> >> >
> >> > My +1 was not contingent on any changes.
> >> >
> >> > The release is fine as-is. Fixing any subset of the release notes,
> >> > minicluster jar, and the organization of LICENSE files within jars
> >> > should not reset the clock on the VOTE. -C
> >> >
> >> > > On Wed, Jan 25, 2017 at 11:14 AM, Karthik Kambatla <
> >> ka...@cloudera.com>
> >> > > wrote:
> >> > >
> >> > >> Thanks for driving the alphas, Andrew. I don't see the need to
> >> restart
> >> > the
> >> > >> vote and I feel it is okay to fix the minor issues before
> releasing.
> >> > >>
> >> > >> +1 (binding). Downloaded source, stood up a pseudo-distributed
> >> cluster
> >> > >> with FairScheduler, ran example jobs, and played around with the
> UI.
> >> > >>
> >> > >> Thanks
> >> > >> Karthik
> >> > >>
> >> > >>
> >> > >> On Fri, Jan 20, 2017 at 2:36 PM, Andrew Wang <
> >> andrew.w...@cloudera.com>
> >> > >> wrote:
> >> > >>
> >> > >>> Hi all,
> >> > >>>
> >> > >>> With heartfelt thanks to many contributors, the RC0 for
> >> 3.0.0-alpha2 is
> >> > >>> ready.
> >> > >>>
> >> > >>> 3.0.0-alpha2 is the second alpha in the planned 3.0.0 release line
> >> > leading
> >> > >>> up to a 3.0.0 GA. It comprises 857 fixes, improvements, and new
> >> > features
> >> > >>> since alpha1 was released on September 3rd, 2016.
> >> > >>>
> >> > >>> More information about the 3.0.0 release plan can be found here:
> >> > >>>
> >> > >>> https://cwiki.apache.org/confluence/display/HADOOP/
> >> > Hadoop+3.0.0+release
> >> > >>>
> >> > >>> The artifacts can be found here:
> >> > >>>
> >> > >>> http://home.apache.org/~wang/3.0.0-alpha2-RC0/
> >> > >>>
> >> > >>> This vote will run 5 days, ending on 01/25/2017 at 2PM pacific.
> >> > >>>
> >> > >>> I ran basic validation with a local pseudo cluster and a Pi job.
> RAT
> >> > >>> output
> >> > >>> was clean.
> >> > >>>
> >> > >>> My +1 to start.
> >> > >>>
> >> > >>> Thanks,
> >> > >>> Andrew
> >> > >>>
> >> > >>
> >> > >>
> >> >
> >>
> >
> >
>

Re: [VOTE] Release cadence and EOL

[Sangjin Lee]

I'm going to stop the vote and go back to the discussion. It shouldn't be a
big surprise given the reservation we have so far.

I do hope there will be some actionable outcome as a result of that
discussion.

Regards,
Sangjin

On Mon, Jan 23, 2017 at 8:17 AM, Allen Wittenauer 
wrote:

>
> > On Jan 21, 2017, at 7:08 PM, Karthik Kambatla 
> wrote:
> >
> >   3. RM: some method to madness. Junping, for instance, is trying to roll
> >   a release with 2300 patches. It is a huge time investment. (Thanks
> again,
> >   Junping.) Smaller releases are easier to manage. A target release
> cadence,
> >   coupled with a process that encourages volunteering, IMO would lead to
> more
> >   committers doing releases.
>
>
> In the case of 2.8.0, that's on the original RM and "back port
> fever" that inflicts way too many committers.  2.8.0 has been sitting in a
> separate branch for over a year.  Of *course* it is going to be a
> disaster.  If the original RM had said "I don't have time, someone take
> over" after 3 months of it being left idle or another committer feeling as
> though they could take it or not everyone dumping everything they can in
> every release possible, it wouldn't be nearly as bad.
>
> Not only do we need to encourage volunteering, but we also need to
> encourage people to relinquish control. If the PMC wants to enact a
> cadence, then they also must be willing to step in when an RM is
> unresponsive and request someone else take over.  A message every three
> months saying "Yes, I'm still working on it." doesn't really help anyone,
> including the RM.
>
>
> > To conclude, the biggest value I see is us (the community) agreeing on
> good
> > practices for our releases and work towards that. Writing it down
> somewhere
> > makes it a little more formal like the compatibility stuff, even if it is
> > not enforceable.
>
> So it's exactly like the compatibility stuff. ;)
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release cadence and EOL

[Sangjin Lee]

Thanks for great feedback thus far.

I agree with some that this is more of a guideline than a policy. I also
agree to some extent that this is not very enforceable given the current
practice.

I do differ, though, on whether there is value in something like this if it
is not completely enforceable. I don't think it's such a binary situation,
and do think there is a plenty of middle ground where a good guideline can
save us time and help us a great deal.

If we view this more as a guideline (or a baseline if you will), this can
act as a reference point for discussion for setting up releases as well as
making decisions on stopping maintenance releases. Today, all of these
discussions invariably have to start from scratch with only past
experiences as a guide. We end up spending a decent amount of time just to
get to a common understanding. If we have a baseline to operate from, at
least we can considerably shorten that discussion and be a little more
productive and consistent as a community. I see value in that.

The security patch for the 2.6.x line is a case in point. Without any
guideline, we would start with "What should we do for 2.6.x? Should we
continue to patch it?" With this guideline, the baseline is already "it's
been 2 years since 2.6.0 is released and we should consider stopping
releasing from 2.6.x and encourage users to upgrade to 2.7.x."

What do you think?

On Thu, Jan 19, 2017 at 7:02 PM, Chris Douglas  wrote:

> Sorry, I'd missed the end of the EOL discussion thread.
>
> As several people have pointed out, this is unenforceable. The release
> dates on the front page are a decent signal for liveness... do we need
> something more formal? All these hypothetical situations would be
> decided with more context. The "good reason" clause allows revive a
> release line if two "live" branches straddle a dud, so this proposal
> only commits us to maintain our mistakes. For two years? As Andrew
> points out, while this heuristic usually holds, we're not set up to
> enforce it.
>
> We may want to have an informal policy for security issues, since
> there are known holes in 2.6.x and earlier that aren't going to be
> patched. We need to issue CVEs for those. A policy would simplify
> tracking (e.g., announce vulnerabilities no more than a month after a
> fix is available in a later release), so we don't wait indefinitely to
> announce. Additionally, creating a JIRA and flagging the release on
> the download page would be ample warning.
>

Actually the decision on security issues is a pretty strong indicator of
our desire for EOL. If we say we do not want to patch that line for
security vulnerability, then there would be even *less* rationale for
fixing any other issue on that line. So the decision to stop backporting
security patches is a sufficient indication of EOL in my mind.


>
> We can still embargo security flaws if someone asks (to give them time
> time to implement a fix and call a vote). If there's nothing else in
> the release, then we're effectively announcing it. In those cases, we
> call a vote on private@ (cc: security@). -C
>
>
> On Thu, Jan 19, 2017 at 1:30 PM, Andrew Wang 
> wrote:
> > I don't think the motivation here is vendor play or taking away power
> from
> > committers. Having a regular release cadence helps our users understand
> > when a feature will ship so they can plan their upgrades. Having an EOL
> > policy and a minimum support period helps users choose a release line,
> and
> > understand when they will need to upgrade.
> >
> > In the earlier thread, we discussed how these are not rules, but
> > guidelines. There's a lot of flexibility if someone wants to keep
> > maintaining a release line (particularly if they are willing to do the
> > backporting work). More power to them; more releases are a good thing for
> > the project.
> >
> > My main concern (which I raised on the earlier thread) is that without
> > significant improvements to the release process and upstream integration
> > testing, it's unlikely we'll actually ship more releases. Too often,
> > branches are simply not in a releaseable state, or they have latent
> blocker
> > bugs due to a lack of testing. This is what we've been struggling with on
> > both the 2.8.x and 3.0.0-x release lines.
> >
> > So, in the abstract, I'm +1 on having a published policy on release
> cadence
> > and EOL. This information helps users.
> >
> > However, I don't think we're ready to actually execute on this policy for
> > the above reasons. This leaves me ambivalent overall, perhaps -0 since
> > publishing a policy we don't follow is more confusing to users.
> >
> > My 2c,
> > Andrew
> >
> >
> >
> > On Thu, Jan 19, 2017 at 12:28 PM, Arpit Agarwal <
> aagar...@hortonworks.com>
> > wrote:
> >
> >> The ASF release policy says releases may not be vetoed [1] so the EOL
> >> policy sounds unenforceable. Not sure a release cadence is enforceable
> >> either since Release Managers are 

Re: Planning for 3.0.0-alpha2

[Sangjin Lee]

It looks like hadoop-cloud-storage-project was missed in the version set?

On Thu, Jan 19, 2017 at 4:19 PM, Andrew Wang 
wrote:

> I've branched branch-3.0.0-alpha2 and moved out the target versions except
> for our last blocker to a new 3.0.0-alpha3 version.
>
> Business can continue as usual, please just set target/fix versions of
> 3.0.0-alpha3 now instead of 3.0.0-alpha2.
>
> Thanks,
> Andrew
>
> On Thu, Jan 19, 2017 at 3:52 PM, Andrew Wang 
> wrote:
>
> > Heads up that I'm branching for 3.0.0-alpha2 and moving out targets
> > versions. We're waiting on one last blocker which is in final stages of
> > review.
> >
> > Best,
> > Andrew
> >
> > On Wed, Jan 4, 2017 at 11:15 AM, Andrew Wang 
> > wrote:
> >
> >> Hi folks,
> >>
> >> Thanks to the hard work of many contributors, we've been steadily
> burning
> >> down alpha2 blockers. There are only 4 remaining, three of which are PA
> and
> >> likely to be committed shortly.
> >>
> >> Once those three go in (hopefully this week), I'm going to cut the
> alpha2
> >> branch and wait for that last blocker. Normal development activity can
> >> continue on trunk and branch-2, and doesn't need to be committed to the
> >> alpha2 branch.
> >>
> >> Since we still have some significant code to wrap up (Tomcat->Jetty
> >> conversion, EC work, rolling upgrade compatibility), it's likely there
> will
> >> also be an alpha3 before we freeze for beta1. I've updated the Hadoop 3
> >> wiki page [1] to reflect this.
> >>
> >> Thanks,
> >> Andrew
> >>
> >> [1]: https://cwiki.apache.org/confluence/display/HADOOP/Hado
> >> op+3.0.0+release
> >>
> >> On Mon, Oct 17, 2016 at 1:57 PM, Andrew Wang 
> >> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> It's been a month since 3.0.0-alpha1, and we've been incorporating
> fixes
> >>> based on downstream feedback. Thus, it's getting to be time for
> >>> 3.0.0-alpha2. I'm using this JIRA query to track open issues:
> >>>
> >>> https://issues.apache.org/jira/issues/?jql=project%20in%20(H
> >>> ADOOP%2C%20HDFS%2C%20MAPREDUCE%2C%20YARN)%20AND%20%22Target%
> >>> 20Version%2Fs%22%20in%20(3.0.0-alpha2%2C%203.0.0-beta1%2C%
> >>> 202.8.0)%20AND%20statusCategory%20not%20in%20(Complete)%
> >>> 20ORDER%20BY%20priority
> >>>
> >>> If alpha2 goes well, we can declare feature freeze, cut branch-3, and
> >>> move onto beta1. My plan for the 3.0.0 release timeline looks like
> this:
> >>>
> >>> * alpha2 in early November
> >>> * beta1 in early Jan
> >>> * GA in early March
> >>>
> >>> I'd appreciate everyone's help in resolving blocker and critical issues
> >>> on the above JIRA search.
> >>>
> >>> Thanks,
> >>> Andrew
> >>>
> >>
> >>
> >
>

Re: [VOTE] Release cadence and EOL

[Sangjin Lee]

Thanks for correcting me! I left out a sentence by mistake. :)

To correct the proposal we're voting for:

A minor release on the latest major line should be every 6 months, and a
maintenance release on a minor release (as there may be concurrently
maintained minor releases) every 2 months.

A minor release line is end-of-lifed 2 years after it is released or there
are 2 newer minor releases, whichever is sooner. The community reserves the
right to extend or shorten the life of a release line if there is a good
reason to do so.

Sorry for the snafu.

Regards,
Sangjin

On Tue, Jan 17, 2017 at 9:58 AM, Daniel Templeton <dan...@cloudera.com>
wrote:

> Thanks for driving this, Sangjin. Quick question, though: the subject line
> is "Release cadence and EOL," but I don't see anything about cadence in the
> proposal.  Did I miss something?
>
> Daniel
>
>
> On 1/17/17 8:35 AM, Sangjin Lee wrote:
>
>> Following up on the discussion thread on this topic (
>> https://s.apache.org/eFOf), I'd like to put the proposal for a vote for
>> the
>> release cadence and EOL. The proposal is as follows:
>>
>> "A minor release line is end-of-lifed 2 years after it is released or
>> there
>> are 2 newer minor releases, whichever is sooner. The community reserves
>> the
>> right to extend or shorten the life of a release line if there is a good
>> reason to do so."
>>
>> This also entails that we the Hadoop community commit to following this
>> practice and solving challenges to make it possible. Andrew Wang laid out
>> some of those challenges and what can be done in the discussion thread
>> mentioned above.
>>
>> I'll set the voting period to 7 days. I understand a majority rule would
>> apply in this case. Your vote is greatly appreciated, and so are
>> suggestions!
>>
>> Thanks,
>> Sangjin
>>
>>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[VOTE] Release cadence and EOL

[Sangjin Lee]

Following up on the discussion thread on this topic (
https://s.apache.org/eFOf), I'd like to put the proposal for a vote for the
release cadence and EOL. The proposal is as follows:

"A minor release line is end-of-lifed 2 years after it is released or there
are 2 newer minor releases, whichever is sooner. The community reserves the
right to extend or shorten the life of a release line if there is a good
reason to do so."

This also entails that we the Hadoop community commit to following this
practice and solving challenges to make it possible. Andrew Wang laid out
some of those challenges and what can be done in the discussion thread
mentioned above.

I'll set the voting period to 7 days. I understand a majority rule would
apply in this case. Your vote is greatly appreciated, and so are
suggestions!

Thanks,
Sangjin

Re: [DISCUSS] Release cadence and EOL

[Sangjin Lee]

Happy new year!

I think this topic has aged quite a bit in the discussion thread. Should we
take it to a vote? Do we need additional discussions?

Regards,
Sangjin

On Wed, Nov 9, 2016 at 11:11 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> Fair points, Sangjin and Andrew.
>
> To get the ball rolling on this, I am willing to try the proposed policy.
>
> On Fri, Nov 4, 2016 at 12:09 PM, Andrew Wang <andrew.w...@cloudera.com>
> wrote:
>
> > I'm certainly willing to try this policy. There's definitely room for
> > improvement when it comes to streamlining the release process. The
> > create-release script that Allen wrote helps, but there are still a lot
> of
> > manual steps in HowToRelease for staging and publishing a release.
> >
> > Another perennial problem is reconciling git log with the changes and
> > release notes and JIRA information. I think each RM has written their own
> > scripts for this, but it could probably be automated into a Jenkins
> report.
> >
> > And the final problem is that branches are often not in a releasable
> > state. This is because we don't have any upstream integration testing.
> For
> > instance, testing with 3.0.0-alpha1 has found a number of latent
> > incompatibilities in the 2.8.0 branch. If we want to meaningfully speed
> up
> > the minor release cycle, continuous integration testing is a must.
> >
> > Best,
> > Andrew
> >
> > On Fri, Nov 4, 2016 at 10:33 AM, Sangjin Lee <sj...@apache.org> wrote:
> >
> >> Thanks for your thoughts and more data points Andrew.
> >>
> >> I share your concern that the proposal may be more aggressive than what
> >> we have been able to accomplish so far. I'd like to hear from the
> community
> >> what is a desirable release cadence which is still within the realm of
> the
> >> possible.
> >>
> >> The EOL policy can also be a bit of a forcing function. By having a
> >> defined EOL, hopefully it would prod the community to move faster with
> >> releases. Of course, automating releases and testing should help.
> >>
> >>
> >> On Tue, Nov 1, 2016 at 4:31 PM, Andrew Wang <andrew.w...@cloudera.com>
> >> wrote:
> >>
> >>> Thanks for pushing on this Sangjin. The proposal sounds reasonable.
> >>>
> >>> However, for it to have teeth, we need to be *very* disciplined about
> the
> >>> release cadence. Looking at our release history, we've done 4
> maintenance
> >>> releases in 2016 and no minor releases. 2015 had 4 maintenance and 1
> >>> minor
> >>> release. The proposal advocates for 12 maintenance releases and 2
> minors
> >>> per year, or about 3.5x more releases than we've historically done. I
> >>> think
> >>> achieving this will require significantly streamlining our release and
> >>> testing process.
> >>>
> >>> For some data points, here are a few EOL lifecycles for some major
> >>> projects. They talk about support in terms of time (not number of
> >>> releases), and release on a cadence.
> >>>
> >>> Ubuntu maintains LTS for 5 years:
> >>> https://www.ubuntu.com/info/release-end-of-life
> >>>
> >>> Linux LTS kernels have EOLs ranging from 2 to 6 years, though it seems
> >>> only
> >>> one has actually ever been EOL'd:
> >>> https://www.kernel.org/category/releases.html
> >>>
> >>> Mesos supports minor releases for 6 months, with a new minor every 2
> >>> months:
> >>> http://mesos.apache.org/documentation/latest/versioning/
> >>>
> >>> Eclipse maintains each minor for ~9 months before moving onto a new
> >>> minor:
> >>> http://stackoverflow.com/questions/35997352/how-to-determine
> >>> -end-of-life-for-eclipse-versions
> >>>
> >>>
> >>>
> >>> On Fri, Oct 28, 2016 at 10:55 AM, Sangjin Lee <sj...@apache.org>
> wrote:
> >>>
> >>> > Reviving an old thread. I think we had a fairly concrete proposal on
> >>> the
> >>> > table that we can vote for.
> >>> >
> >>> > The proposal is a minor release on the latest major line every 6
> >>> months,
> >>> > and a maintenance release on a minor release (as there may be
> >>> concurrently
> >>> > maintained minor releases) every 2 months.
> >>> >
> >>> > A minor release line 

Re: Updated 2.8.0-SNAPSHOT artifact

[Sangjin Lee]

We could consider porting a limited number of critical fixes to branch-2.8.
We'd need to be judicious about it so as not to prolong that cycle, however.


Sangjin

On Thu, Nov 10, 2016 at 6:16 AM, Eric Payne <erichadoo...@yahoo.com> wrote:

> How do we come to a resolution regarding whether or not re-cut branch-2.8
> or release it as it is (after fixing blockers)?
>
> There are some things in branch-2 that I would like to pull back into
> branch-2.8, so a resolution to this question will affect how I proceed.
>
> Thanks,
> -Eric
>
> --
> *From:* Karthik Kambatla <ka...@cloudera.com>
> *To:* Ming Ma <min...@twitter.com>
> *Cc:* Sangjin Lee <sjl...@gmail.com>; Jason Lowe <jl...@yahoo-inc.com>;
> Akira Ajisaka <ajisa...@oss.nttdata.co.jp>; Brahma Reddy Battula <
> brahmareddy.batt...@huawei.com>; Vinod Kumar Vavilapalli <
> vino...@apache.org>; "common-...@hadoop.apache.org" <
> common-...@hadoop.apache.org>; "hdfs-...@hadoop.apache.org" <
> hdfs-...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <
> mapreduce-dev@hadoop.apache.org>; "yarn-...@hadoop.apache.org" <
> yarn-...@hadoop.apache.org>
> *Sent:* Thursday, November 10, 2016 1:56 AM
>
> *Subject:* Re: Updated 2.8.0-SNAPSHOT artifact
>
> If there is interest in releasing off of branch-2.8, we should definitely
> do that. As Sangjin mentioned, there might be value in doing 2.9 off
> branch-2 too.
>
> How do we go about maintenance releases along those minor lines, and when
> would we discontinue 2.6.x/2.7.x releases?
>
> On Wed, Nov 9, 2016 at 12:06 PM, Ming Ma <min...@twitter.com> wrote:
>
> > I would also prefer releasing current 2.8 branch sooner. There are
> several
> > incomplete features in branch-2 such as YARN-914 and HDFS-7877 that are
> > better served if we can complete them in the next major release. Letting
> > them span across multiple releases might not be desirable as there could
> be
> > some potential compatibility issues involved. Therefore if we recut 2.8
> it
> > means we have to work on those items before the new 2.8 is released which
> > could cause major delay on the schedule.
> >
> > On Mon, Nov 7, 2016 at 10:37 AM, Sangjin Lee <sjl...@gmail.com> wrote:
> >
> >> +1. Resetting the 2.8 effort and the branch at this point may be
> >> counter-productive. IMO we should focus on resolving the remaining
> >> blockers
> >> and getting it out the door. I also think that we should seriously
> >> consider
> >> 2.9 as well, as a fairly large number of changes have accumulated in
> >> branch-2 (over branch-2.8).
> >>
> >>
> >> Sangjin
> >>
> >> On Fri, Nov 4, 2016 at 3:38 PM, Jason Lowe <jl...@yahoo-inc.com.invalid
> >
> >> wrote:
> >>
> >> > At this point my preference would be to do the most expeditious thing
> to
> >> > release 2.8, whether that's sticking with the branch-2.8 we have today
> >> or
> >> > re-cutting it on branch-2.  Doing a quick JIRA query, there's been
> >> almost
> >> > 2,400 JIRAs resolved in 2.8.0 (1).  For many of them, it's well-past
> >> time
> >> > they saw a release vehicle.  If re-cutting the branch means we have to
> >> wrap
> >> > up a few extra things that are still in-progress on branch-2 or add a
> >> few
> >> > more blockers to the list before we release then I'd rather stay where
> >> > we're at and ship it ASAP.
> >> >
> >> > Jason
> >> > (1) https://issues.apache.org/jira/issues/?jql=project%20in%
> >> > 20%28hadoop%2C%20yarn%2C%20mapreduce%2C%20hdfs%29%
> >> > 20and%20resolution%20%3D%20Fixed%20and%20fixVersion%20%3D%202.8.0
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >On Tuesday, October 25, 2016 5:31 PM, Karthik Kambatla <
> >> > ka...@cloudera.com> wrote:
> >> >
> >> >
> >> >  Is there value in releasing current branch-2.8? Aren't we better off
> >> > re-cutting the branch off of branch-2?
> >> >
> >> > On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka <
> >> > ajisa...@oss.nttdata.co.jp>
> >> > wrote:
> >> >
> >> > > It's almost a year since branch-2.8 has cut.
> >> > > I'm thinking we need to release 2.8.0 ASAP.
> >> > >
> >> > > According to the following list, the

Re: Updated 2.8.0-SNAPSHOT artifact

[Sangjin Lee]

On Wed, Nov 9, 2016 at 11:56 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> If there is interest in releasing off of branch-2.8, we should definitely
> do that. As Sangjin mentioned, there might be value in doing 2.9 off
> branch-2 too.
>
> How do we go about maintenance releases along those minor lines, and when
> would we discontinue 2.6.x/2.7.x releases?
>

Per proposal on the release cadence and EOL discussed in another thread,
once 2.8 is released probably the 2.6.x line would be EOLed. And that
doesn't sound unreasonable.


>
> On Wed, Nov 9, 2016 at 12:06 PM, Ming Ma <min...@twitter.com> wrote:
>
>> I would also prefer releasing current 2.8 branch sooner. There are
>> several incomplete features in branch-2 such as YARN-914 and HDFS-7877 that
>> are better served if we can complete them in the next major release.
>> Letting them span across multiple releases might not be desirable as there
>> could be some potential compatibility issues involved. Therefore if we
>> recut 2.8 it means we have to work on those items before the new 2.8 is
>> released which could cause major delay on the schedule.
>>
>> On Mon, Nov 7, 2016 at 10:37 AM, Sangjin Lee <sjl...@gmail.com> wrote:
>>
>>> +1. Resetting the 2.8 effort and the branch at this point may be
>>> counter-productive. IMO we should focus on resolving the remaining
>>> blockers
>>> and getting it out the door. I also think that we should seriously
>>> consider
>>> 2.9 as well, as a fairly large number of changes have accumulated in
>>> branch-2 (over branch-2.8).
>>>
>>>
>>> Sangjin
>>>
>>> On Fri, Nov 4, 2016 at 3:38 PM, Jason Lowe <jl...@yahoo-inc.com.invalid>
>>> wrote:
>>>
>>> > At this point my preference would be to do the most expeditious thing
>>> to
>>> > release 2.8, whether that's sticking with the branch-2.8 we have today
>>> or
>>> > re-cutting it on branch-2.  Doing a quick JIRA query, there's been
>>> almost
>>> > 2,400 JIRAs resolved in 2.8.0 (1).  For many of them, it's well-past
>>> time
>>> > they saw a release vehicle.  If re-cutting the branch means we have to
>>> wrap
>>> > up a few extra things that are still in-progress on branch-2 or add a
>>> few
>>> > more blockers to the list before we release then I'd rather stay where
>>> > we're at and ship it ASAP.
>>> >
>>> > Jason
>>> > (1) https://issues.apache.org/jira/issues/?jql=project%20in%
>>> > 20%28hadoop%2C%20yarn%2C%20mapreduce%2C%20hdfs%29%
>>> > 20and%20resolution%20%3D%20Fixed%20and%20fixVersion%20%3D%202.8.0
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Tuesday, October 25, 2016 5:31 PM, Karthik Kambatla <
>>> > ka...@cloudera.com> wrote:
>>> >
>>> >
>>> >  Is there value in releasing current branch-2.8? Aren't we better off
>>> > re-cutting the branch off of branch-2?
>>> >
>>> > On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka <
>>> > ajisa...@oss.nttdata.co.jp>
>>> > wrote:
>>> >
>>> > > It's almost a year since branch-2.8 has cut.
>>> > > I'm thinking we need to release 2.8.0 ASAP.
>>> > >
>>> > > According to the following list, there are 5 blocker and 6 critical
>>> > issues.
>>> > > https://issues.apache.org/jira/issues/?filter=12334985
>>> > >
>>> > > Regards,
>>> > > Akira
>>> > >
>>> > >
>>> > > On 10/18/16 10:47, Brahma Reddy Battula wrote:
>>> > >
>>> > >> Hi Vinod,
>>> > >>
>>> > >> Any plan on first RC for branch-2.8 ? I think, it has been long
>>> time.
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >> --Brahma Reddy Battula
>>> > >>
>>> > >> -Original Message-
>>> > >> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org]
>>> > >> Sent: 20 August 2016 00:56
>>> > >> To: Jonathan Eagles
>>> > >> Cc: common-...@hadoop.apache.org
>>> > >> Subject: Re: Updated 2.8.0-SNAPSHOT artifact
>>> > >>
>>> > >> Jon,
>>> > >>
>>> > >> That is around the time when I br

Re: Updated 2.8.0-SNAPSHOT artifact

[Sangjin Lee]

+1. Resetting the 2.8 effort and the branch at this point may be
counter-productive. IMO we should focus on resolving the remaining blockers
and getting it out the door. I also think that we should seriously consider
2.9 as well, as a fairly large number of changes have accumulated in
branch-2 (over branch-2.8).


Sangjin

On Fri, Nov 4, 2016 at 3:38 PM, Jason Lowe 
wrote:

> At this point my preference would be to do the most expeditious thing to
> release 2.8, whether that's sticking with the branch-2.8 we have today or
> re-cutting it on branch-2.  Doing a quick JIRA query, there's been almost
> 2,400 JIRAs resolved in 2.8.0 (1).  For many of them, it's well-past time
> they saw a release vehicle.  If re-cutting the branch means we have to wrap
> up a few extra things that are still in-progress on branch-2 or add a few
> more blockers to the list before we release then I'd rather stay where
> we're at and ship it ASAP.
>
> Jason
> (1) https://issues.apache.org/jira/issues/?jql=project%20in%
> 20%28hadoop%2C%20yarn%2C%20mapreduce%2C%20hdfs%29%
> 20and%20resolution%20%3D%20Fixed%20and%20fixVersion%20%3D%202.8.0
>
>
>
>
>
> On Tuesday, October 25, 2016 5:31 PM, Karthik Kambatla <
> ka...@cloudera.com> wrote:
>
>
>  Is there value in releasing current branch-2.8? Aren't we better off
> re-cutting the branch off of branch-2?
>
> On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka <
> ajisa...@oss.nttdata.co.jp>
> wrote:
>
> > It's almost a year since branch-2.8 has cut.
> > I'm thinking we need to release 2.8.0 ASAP.
> >
> > According to the following list, there are 5 blocker and 6 critical
> issues.
> > https://issues.apache.org/jira/issues/?filter=12334985
> >
> > Regards,
> > Akira
> >
> >
> > On 10/18/16 10:47, Brahma Reddy Battula wrote:
> >
> >> Hi Vinod,
> >>
> >> Any plan on first RC for branch-2.8 ? I think, it has been long time.
> >>
> >>
> >>
> >>
> >> --Brahma Reddy Battula
> >>
> >> -Original Message-
> >> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org]
> >> Sent: 20 August 2016 00:56
> >> To: Jonathan Eagles
> >> Cc: common-...@hadoop.apache.org
> >> Subject: Re: Updated 2.8.0-SNAPSHOT artifact
> >>
> >> Jon,
> >>
> >> That is around the time when I branched 2.8, so I guess you were getting
> >> SNAPSHOT artifacts till then from the branch-2 nightly builds.
> >>
> >> If you need it, we can set up SNAPSHOT builds. Or just wait for the
> first
> >> RC, which is around the corner.
> >>
> >> +Vinod
> >>
> >> On Jul 28, 2016, at 4:27 PM, Jonathan Eagles  wrote:
> >>>
> >>> Latest snapshot is uploaded in Nov 2015, but checkins are still coming
> >>> in quite frequently.
> >>> https://repository.apache.org/content/repositories/snapshots/org/apach
> >>> e/hadoop/hadoop-yarn-api/
> >>>
> >>> Are there any plans to start producing updated SNAPSHOT artifacts for
> >>> current hadoop development lines?
> >>>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>
> >>
> >
> > -
> > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >
> >
>
>
>
>

Re: [DISCUSS] Release cadence and EOL

[Sangjin Lee]

Thanks for your thoughts and more data points Andrew.

I share your concern that the proposal may be more aggressive than what we
have been able to accomplish so far. I'd like to hear from the community
what is a desirable release cadence which is still within the realm of the
possible.

The EOL policy can also be a bit of a forcing function. By having a defined
EOL, hopefully it would prod the community to move faster with releases. Of
course, automating releases and testing should help.


On Tue, Nov 1, 2016 at 4:31 PM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Thanks for pushing on this Sangjin. The proposal sounds reasonable.
>
> However, for it to have teeth, we need to be *very* disciplined about the
> release cadence. Looking at our release history, we've done 4 maintenance
> releases in 2016 and no minor releases. 2015 had 4 maintenance and 1 minor
> release. The proposal advocates for 12 maintenance releases and 2 minors
> per year, or about 3.5x more releases than we've historically done. I think
> achieving this will require significantly streamlining our release and
> testing process.
>
> For some data points, here are a few EOL lifecycles for some major
> projects. They talk about support in terms of time (not number of
> releases), and release on a cadence.
>
> Ubuntu maintains LTS for 5 years:
> https://www.ubuntu.com/info/release-end-of-life
>
> Linux LTS kernels have EOLs ranging from 2 to 6 years, though it seems only
> one has actually ever been EOL'd:
> https://www.kernel.org/category/releases.html
>
> Mesos supports minor releases for 6 months, with a new minor every 2
> months:
> http://mesos.apache.org/documentation/latest/versioning/
>
> Eclipse maintains each minor for ~9 months before moving onto a new minor:
> http://stackoverflow.com/questions/35997352/how-to-
> determine-end-of-life-for-eclipse-versions
>
>
>
> On Fri, Oct 28, 2016 at 10:55 AM, Sangjin Lee <sj...@apache.org> wrote:
>
> > Reviving an old thread. I think we had a fairly concrete proposal on the
> > table that we can vote for.
> >
> > The proposal is a minor release on the latest major line every 6 months,
> > and a maintenance release on a minor release (as there may be
> concurrently
> > maintained minor releases) every 2 months.
> >
> > A minor release line is EOLed 2 years after it is first released or there
> > are 2 newer minor releases, whichever is sooner. The community reserves
> the
> > right to extend or shorten the life of a release line if there is a good
> > reason to do so.
> >
> > Comments? Objections?
> >
> > Regards,
> > Sangjin
> >
> >
> > On Tue, Aug 23, 2016 at 9:33 AM, Karthik Kambatla <ka...@cloudera.com>
> > wrote:
> >
> > >
> > >> Here is just an idea to get started. How about "a minor release line
> is
> > >> EOLed 2 years after it is released or there are 2 newer minor
> releases,
> > >> whichever is sooner. The community reserves the right to extend or
> > shorten
> > >> the life of a release line if there is a good reason to do so."
> > >>
> > >>
> > > Sounds reasonable, especially for our first commitment. For current
> > > releases, this essentially means 2.6.x is maintained until Nov 2016 and
> > Apr
> > > 2017 if 2.8 and 2.9 are not released by those dates.
> > >
> > > IIUC EOL does two things - (1) eases the maintenance cost for
> developers
> > > past EOL, and (2) indicates to the user when they must upgrade by. For
> > the
> > > latter, would users appreciate a specific timeline without any caveats
> > for
> > > number of subsequent minor releases?
> > >
> > > If we were to give folks a specific period for EOL for x.y.z, we should
> > > plan on releasing at least x.y+1.1 by then. 2 years might be a good
> > number
> > > to start with given our current cadence, and adjusted in the future as
> > > needed.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Release cadence and EOL

[Sangjin Lee]

Reviving an old thread. I think we had a fairly concrete proposal on the
table that we can vote for.

The proposal is a minor release on the latest major line every 6 months,
and a maintenance release on a minor release (as there may be concurrently
maintained minor releases) every 2 months.

A minor release line is EOLed 2 years after it is first released or there
are 2 newer minor releases, whichever is sooner. The community reserves the
right to extend or shorten the life of a release line if there is a good
reason to do so.

Comments? Objections?

Regards,
Sangjin


On Tue, Aug 23, 2016 at 9:33 AM, Karthik Kambatla 
wrote:

>
>> Here is just an idea to get started. How about "a minor release line is
>> EOLed 2 years after it is released or there are 2 newer minor releases,
>> whichever is sooner. The community reserves the right to extend or shorten
>> the life of a release line if there is a good reason to do so."
>>
>>
> Sounds reasonable, especially for our first commitment. For current
> releases, this essentially means 2.6.x is maintained until Nov 2016 and Apr
> 2017 if 2.8 and 2.9 are not released by those dates.
>
> IIUC EOL does two things - (1) eases the maintenance cost for developers
> past EOL, and (2) indicates to the user when they must upgrade by. For the
> latter, would users appreciate a specific timeline without any caveats for
> number of subsequent minor releases?
>
> If we were to give folks a specific period for EOL for x.y.z, we should
> plan on releasing at least x.y+1.1 by then. 2 years might be a good number
> to start with given our current cadence, and adjusted in the future as
> needed.
>
>
>

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Sangjin Lee]

Thanks everyone for checking out this release! The vote passes with 15
+1's, 7 of which are binding, and no -1's.

I'll go ahead and wrap up the release, and will send an announcement in the
next day or two.

Regards,
Sangjin

On Mon, Oct 10, 2016 at 6:38 AM, Jason Lowe <jl...@yahoo-inc.com> wrote:

> +1 (binding)
>
> - Verified signatures and digests
> - Built native from source
> - Deployed to a single-node cluster and ran some sample jobs
>
> Jason
>
>
> On Sunday, October 2, 2016 7:13 PM, Sangjin Lee <sj...@apache.org> wrote:
>
>
> Hi folks,
>
> I have pushed a new release candidate (R1) for the Apache Hadoop 2.6.5
> release (the next maintenance release in the 2.6.x release line). RC1
> contains fixes to CHANGES.txt, and is otherwise identical to RC0.
>
> Below are the details of this release candidate:
>
> The RC is available for validation at:
> http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/.
>
> The RC tag in git is release-2.6.5-RC1 and its git commit is
> e8c9fe0b4c252caf2ebf1464220599650f119997.
>
> The maven artifacts are staged via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1050/.
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
>
> Please try the release and vote. The vote will run for the usual 5 days. I
> would greatly appreciate your timely vote. Thanks!
>
> Regards,
> Sangjin
>
>
>

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Sangjin Lee]

I'm casting my vote: +1 (binding)

Regards,
Sangjin

On Fri, Oct 7, 2016 at 3:12 PM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Thanks to Chris and Sangjin for working on this release.
>
> +1 binding
>
> * Verified signatures
> * Built from source tarball
> * Started HDFS and did some basic ops
>
> Thanks,
> Andrew
>
> On Fri, Oct 7, 2016 at 2:50 PM, Wangda Tan <wheele...@gmail.com> wrote:
>
> > Thanks Sangjin for cutting this release!
> >
> > +1 (Binding)
> >
> > - Downloaded binary tar ball and setup a single node cluster.
> > - Submit a few applications and which can successfully run.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Fri, Oct 7, 2016 at 10:33 AM, Zhihai Xu <zhihaixu2...@gmail.com>
> wrote:
> >
> > > Thanks Sangjin for creating release 2.6.5 RC1.
> > >
> > > +1 (non-binding)
> > >
> > > * Downloaded and built from source
> > > * Verified md5 checksums and signature
> > > * Deployed a pseudo cluster
> > > * verified basic HDFS operations and Pi job.
> > > * Did a sanity check for RM and NM UI.
> > >
> > > Thanks
> > > zhihai
> > >
> > > On Fri, Oct 7, 2016 at 8:16 AM, Sangjin Lee <sj...@apache.org> wrote:
> > >
> > > > Thanks Masatake!
> > > >
> > > > Today's the last day for this vote, and I'd like to ask you to try
> out
> > > the
> > > > RC and vote on this today. So far there has been no binding vote.
> > Thanks
> > > > again.
> > > >
> > > > Regards,
> > > > Sangjin
> > > >
> > > > On Fri, Oct 7, 2016 at 6:45 AM, Masatake Iwasaki <
> > > > iwasak...@oss.nttdata.co.jp> wrote:
> > > >
> > > > > +1(non-binding)
> > > > >
> > > > > * verified signature and md5.
> > > > > * built with -Pnative on CentOS6 and OpenJDK7.
> > > > > * built documentation and skimmed the contents.
> > > > > * built rpms by bigtop and ran smoke-tests of hdfs, yarn and
> > mapreduce
> > > on
> > > > > 3-node cluster.
> > > > >
> > > > > Thanks,
> > > > > Masatake Iwasaki
> > > > >
> > > > > On 10/3/16 09:12, Sangjin Lee wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> I have pushed a new release candidate (R1) for the Apache Hadoop
> > 2.6.5
> > > > >> release (the next maintenance release in the 2.6.x release line).
> > RC1
> > > > >> contains fixes to CHANGES.txt, and is otherwise identical to RC0.
> > > > >>
> > > > >> Below are the details of this release candidate:
> > > > >>
> > > > >> The RC is available for validation at:
> > > > >> http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/.
> > > > >>
> > > > >> The RC tag in git is release-2.6.5-RC1 and its git commit is
> > > > >> e8c9fe0b4c252caf2ebf1464220599650f119997.
> > > > >>
> > > > >> The maven artifacts are staged via repository.apache.org at:
> > > > >> https://repository.apache.org/content/repositories/
> > > > orgapachehadoop-1050/.
> > > > >>
> > > > >> You can find my public key at
> > > > >> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
> > > > >>
> > > > >> Please try the release and vote. The vote will run for the usual 5
> > > > days. I
> > > > >> would greatly appreciate your timely vote. Thanks!
> > > > >>
> > > > >> Regards,
> > > > >> Sangjin
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Sangjin Lee]

Thanks Masatake!

Today's the last day for this vote, and I'd like to ask you to try out the
RC and vote on this today. So far there has been no binding vote. Thanks
again.

Regards,
Sangjin

On Fri, Oct 7, 2016 at 6:45 AM, Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> +1(non-binding)
>
> * verified signature and md5.
> * built with -Pnative on CentOS6 and OpenJDK7.
> * built documentation and skimmed the contents.
> * built rpms by bigtop and ran smoke-tests of hdfs, yarn and mapreduce on
> 3-node cluster.
>
> Thanks,
> Masatake Iwasaki
>
> On 10/3/16 09:12, Sangjin Lee wrote:
>
>> Hi folks,
>>
>> I have pushed a new release candidate (R1) for the Apache Hadoop 2.6.5
>> release (the next maintenance release in the 2.6.x release line). RC1
>> contains fixes to CHANGES.txt, and is otherwise identical to RC0.
>>
>> Below are the details of this release candidate:
>>
>> The RC is available for validation at:
>> http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/.
>>
>> The RC tag in git is release-2.6.5-RC1 and its git commit is
>> e8c9fe0b4c252caf2ebf1464220599650f119997.
>>
>> The maven artifacts are staged via repository.apache.org at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1050/.
>>
>> You can find my public key at
>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
>>
>> Please try the release and vote. The vote will run for the usual 5 days. I
>> would greatly appreciate your timely vote. Thanks!
>>
>> Regards,
>> Sangjin
>>
>>
>

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Sangjin Lee]

I think it's a good idea. We can still have the underlying data (in
*-version.properties) but we can consider removing the date and the build
person from the web UI. I'll file a JIRA to consider the idea.

I would greatly appreciate it if you folks can take a little time to take
the RC1 for a spin and vote on it. Tomorrow will be the conclusion of the
vote, and I haven't received many votes yet. Thanks!

Sangjin

On Thu, Oct 6, 2016 at 3:13 PM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> I don't think it's a blocker, though I wonder what the point of that date
> and attribution are on the WebUI.
>
> Maybe file a follow-on JIRA to remove it for the future?
>
> On Thu, Oct 6, 2016 at 3:09 PM, Sangjin Lee <sj...@apache.org> wrote:
>
> > I looked into building it on jenkins earlier, but it appears that this
> > jenkins job is busted (at least for 2.6.x):
> > https://builds.apache.org/job/HADOOP2_Release_Artifacts_Builder/
> >
> > I don't have access to the job configuration, so I'm unable to fix it
> atm.
> >
> > That said, I do think Allen has a point. It strikes me that the release
> > manager building it under his/her own control shouldn't be a problem, if
> > not better. Also, the how-to-release wiki currently seems to suggest
> > building it locally is fine (http://wiki.apache.org/hadoop/HowToRelease
> ).
> >
> > IMO I don't think this is a blocker. Thoughts?
> >
> > Thanks,
> > Sangjin
> >
> > On Thu, Oct 6, 2016 at 1:39 PM, Akira Ajisaka <
> ajisa...@oss.nttdata.co.jp>
> > wrote:
> >
> > > > It wasn't 'renamed' to jenkins, prior releases were actually built by
> > > and on the Jenkins infrastructure. Which was a very very bad idea:
> it's
> > > insecure and pretty much against ASF policy.
> > >
> > > Sorry for the confusion. I should not have used the word 'rename'.
> > > What I meant is that "would you change the name to 'jenkins' by using
> the
> > > Jenkins infra?"
> > >
> > > Regards,
> > > Akira
> > >
> > >
> > > On 10/7/16 03:04, Allen Wittenauer wrote:
> > >
> > >>
> > >> On Oct 5, 2016, at 10:35 PM, Akira Ajisaka <
> ajisa...@oss.nttdata.co.jp>
> > >>> wrote:
> > >>> Can we rename it?
> > >>>
> > >>> AFAIK, hadoop releases were built by hortonmu in 2014 and was renamed
> > to
> > >>> jenkins.
> > >>>
> > >>
> > >> That's not how that works.
> > >>
> > >> It's literally storing the id of the person who built the
> > >> classes.  It wasn't 'renamed' to jenkins, prior releases were actually
> > >> built by and on the Jenkins infrastructure. Which was a very very bad
> > >> idea:  it's insecure and pretty much against ASF policy.
> > >> -
> > >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >>
> > >>
> > >
> > > -
> > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Sangjin Lee]

I looked into building it on jenkins earlier, but it appears that this
jenkins job is busted (at least for 2.6.x):
https://builds.apache.org/job/HADOOP2_Release_Artifacts_Builder/

I don't have access to the job configuration, so I'm unable to fix it atm.

That said, I do think Allen has a point. It strikes me that the release
manager building it under his/her own control shouldn't be a problem, if
not better. Also, the how-to-release wiki currently seems to suggest
building it locally is fine (http://wiki.apache.org/hadoop/HowToRelease).

IMO I don't think this is a blocker. Thoughts?

Thanks,
Sangjin

On Thu, Oct 6, 2016 at 1:39 PM, Akira Ajisaka 
wrote:

> > It wasn't 'renamed' to jenkins, prior releases were actually built by
> and on the Jenkins infrastructure. Which was a very very bad idea:  it's
> insecure and pretty much against ASF policy.
>
> Sorry for the confusion. I should not have used the word 'rename'.
> What I meant is that "would you change the name to 'jenkins' by using the
> Jenkins infra?"
>
> Regards,
> Akira
>
>
> On 10/7/16 03:04, Allen Wittenauer wrote:
>
>>
>> On Oct 5, 2016, at 10:35 PM, Akira Ajisaka 
>>> wrote:
>>> Can we rename it?
>>>
>>> AFAIK, hadoop releases were built by hortonmu in 2014 and was renamed to
>>> jenkins.
>>>
>>
>> That's not how that works.
>>
>> It's literally storing the id of the person who built the
>> classes.  It wasn't 'renamed' to jenkins, prior releases were actually
>> built by and on the Jenkins infrastructure. Which was a very very bad
>> idea:  it's insecure and pretty much against ASF policy.
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Sangjin Lee]

Hi folks,

I have pushed a new release candidate (R1) for the Apache Hadoop 2.6.5
release (the next maintenance release in the 2.6.x release line). RC1
contains fixes to CHANGES.txt, and is otherwise identical to RC0.

Below are the details of this release candidate:

The RC is available for validation at:
http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/.

The RC tag in git is release-2.6.5-RC1 and its git commit is
e8c9fe0b4c252caf2ebf1464220599650f119997.

The maven artifacts are staged via repository.apache.org at:
https://repository.apache.org/content/repositories/orgapachehadoop-1050/.

You can find my public key at
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.

Please try the release and vote. The vote will run for the usual 5 days. I
would greatly appreciate your timely vote. Thanks!

Regards,
Sangjin

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC0)

[Sangjin Lee]

I am canceling the vote for RC0, and will put up RC1 with the fixed
CHANGES.txt files soon. Thanks everyone!

Sangjin

On Sun, Oct 2, 2016 at 2:58 AM, Akira Ajisaka <aajis...@apache.org> wrote:

> -0 (binding)
>
> - Downloaded source tarball and binary tarball
> - Verified signatures and checksums
> - Compiled successfully with both JDK6 and JDK7
> - Compiled and built a single node cluster
> - Compiled Hive 1.2.1 and Tez 0.7.1/0.8.4 using Hadoop 2.6.5 pom
> successfully
> - Ran some Hive on Tez/MRv2 queries successfully
>
> As John and Brahma noticed, there are some missing entries in CHANGES.txt.
> When upgrading to 2.6.5, most of admins look at the change logs and see
> what bugs are fixed. Therefore I'm thinking the change logs are important
> and they should be fixed.
>
> Thanks,
> Akira
>
>
> On 9/28/16 05:28, Sangjin Lee wrote:
>
>> Hi folks,
>>
>> I have created a release candidate RC0 for the Apache Hadoop 2.6.5 release
>> (the next maintenance release in the 2.6.x release line). Below are the
>> details of this release candidate:
>>
>> The RC is available for validation at:
>> http://home.apache.org/~sjlee/hadoop-2.6.5-RC0/.
>>
>> The RC tag in git is release-2.6.5-RC0 and its git commit is
>> 6939fc935fba5651fdb33386d88aeb8e875cf27a.
>>
>> The maven artifacts are staged via repository.apache.org at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1048/.
>>
>> You can find my public key at
>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
>>
>> Please try the release and vote. The vote will run for the usual 5 days.
>> Huge thanks to Chris Trezzo for spearheading the release management and
>> doing all the work!
>>
>> Thanks,
>> Sangjin
>>
>>
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC0)

[Sangjin Lee]

I took a look at the commits that are part of 2.6.5, and sadly I count a
total of 18 of them that either are missing the corresponding entries or
have incorrect release versions in CHANGES.txt. And I'm not even thinking
about possible missing entries in earlier releases. :(

I think us committers need to pay greater attention to detail when we port
fixes to 2.6.x and 2.7.x where CHANGES.txt are still used. The rate of
missing entries in CHANGES.txt is too high.

Due to this high number of missing entries, I am leaning towards cutting
another RC where at least 2.6.5 entries all fixed. Let me know what you all
think (whether this is something that should stop the RC).

Regards,
Sangjin

On Sat, Oct 1, 2016 at 12:46 PM, Sangjin Lee <sj...@apache.org> wrote:

> Thanks John and Brahma for reporting issues with CHANGES.txt.
>
> IMO, we can move ahead with the current RC0 and address these issues in a
> commit after the release, but I'm not against cutting another RC to fix
> CHANGES.txt. What do you think?
>
> Thanks,
> Sangjin
>
> On Fri, Sep 30, 2016 at 9:25 PM, Brahma Reddy Battula <
> brahmareddy.batt...@hotmail.com> wrote:
>
>> Thanks Sangjin!
>>
>>
>> +1 ( non- binding)
>>
>>
>> --Downloaded the source and complied
>>
>> --Installed HA cluster
>>
>> --Verified basic fsshell commands
>>
>> ---Did the regression on issues which is handled by me
>>
>> --Ran pi,terasort Slive jobs, all works fine.
>>
>>
>> Happy to see HDFS-9530.
>>
>>
>> Read through the commit log,Seems to be following commits are missed in
>> changes.txt.
>>
>>
>> HDFS-10653
>>
>> ,HADOOP-13290
>>
>> ,HDFS-10544,
>>
>> HADOOP-13255
>>
>> ,HADOOP-13189
>>
>>
>>
>>
>> --Brahma Reddy Battula
>>
>>
>> --
>> *From:* sjl...@gmail.com <sjl...@gmail.com> on behalf of Sangjin Lee <
>> sj...@apache.org>
>> *Sent:* Wednesday, September 28, 2016 1:58 AM
>> *To:* common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
>> yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> *Subject:* [VOTE] Release Apache Hadoop 2.6.5 (RC0)
>>
>> Hi folks,
>>
>> I have created a release candidate RC0 for the Apache Hadoop 2.6.5 release
>> (the next maintenance release in the 2.6.x release line). Below are the
>> details of this release candidate:
>>
>> The RC is available for validation at:
>> http://home.apache.org/~sjlee/hadoop-2.6.5-RC0/.
>>
>> The RC tag in git is release-2.6.5-RC0 and its git commit is
>> 6939fc935fba5651fdb33386d88aeb8e875cf27a.
>>
>> The maven artifacts are staged via repository.apache.org at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1048/.
>>
>> You can find my public key at
>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
>>
>> Please try the release and vote. The vote will run for the usual 5 days.
>> Huge thanks to Chris Trezzo for spearheading the release management and
>> doing all the work!
>>
>> Thanks,
>> Sangjin
>>
>
>

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC0)

[Sangjin Lee]

Thanks John and Brahma for reporting issues with CHANGES.txt.

IMO, we can move ahead with the current RC0 and address these issues in a
commit after the release, but I'm not against cutting another RC to fix
CHANGES.txt. What do you think?

Thanks,
Sangjin

On Fri, Sep 30, 2016 at 9:25 PM, Brahma Reddy Battula <
brahmareddy.batt...@hotmail.com> wrote:

> Thanks Sangjin!
>
>
> +1 ( non- binding)
>
>
> --Downloaded the source and complied
>
> --Installed HA cluster
>
> --Verified basic fsshell commands
>
> ---Did the regression on issues which is handled by me
>
> --Ran pi,terasort Slive jobs, all works fine.
>
>
> Happy to see HDFS-9530.
>
>
> Read through the commit log,Seems to be following commits are missed in
> changes.txt.
>
>
> HDFS-10653
>
> ,HADOOP-13290
>
> ,HDFS-10544,
>
> HADOOP-13255
>
> ,HADOOP-13189
>
>
>
>
> --Brahma Reddy Battula
>
>
> --
> *From:* sjl...@gmail.com <sjl...@gmail.com> on behalf of Sangjin Lee <
> sj...@apache.org>
> *Sent:* Wednesday, September 28, 2016 1:58 AM
> *To:* common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> *Subject:* [VOTE] Release Apache Hadoop 2.6.5 (RC0)
>
> Hi folks,
>
> I have created a release candidate RC0 for the Apache Hadoop 2.6.5 release
> (the next maintenance release in the 2.6.x release line). Below are the
> details of this release candidate:
>
> The RC is available for validation at:
> http://home.apache.org/~sjlee/hadoop-2.6.5-RC0/.
>
> The RC tag in git is release-2.6.5-RC0 and its git commit is
> 6939fc935fba5651fdb33386d88aeb8e875cf27a.
>
> The maven artifacts are staged via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1048/.
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
>
> Please try the release and vote. The vote will run for the usual 5 days.
> Huge thanks to Chris Trezzo for spearheading the release management and
> doing all the work!
>
> Thanks,
> Sangjin
>

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC0)

[Sangjin Lee]

Thanks Ming!

I know it's getting late on Friday, but I'd like to ask you folks to take a
little time to check out the release candidate and vote on it. That would
be fantastic. Thanks!

Sangjin

On Fri, Sep 30, 2016 at 3:53 PM Ming Ma <min...@twitter.com.invalid> wrote:

> +1
>
> Successfully compiled a standalone HDFS app using the 2.6.5 jars extracted
> from the release tar.gz.
>
> On Thu, Sep 29, 2016 at 10:33 AM, Chris Trezzo <ctre...@gmail.com> wrote:
>
> > +1
> >
> > Thanks Sangjin!
> >
> > 1. Verified md5 checksums and signature on src, and release tar.gz.
> > 2. Built from source.
> > 3. Started up a pseudo distributed cluster.
> > 4. Successfully ran a PI job.
> > 5. Ran the balancer.
> > 6. Inspected UI for RM, NN, JobHistory.
> >
> > On Tue, Sep 27, 2016 at 4:11 PM, Lei Xu <l...@cloudera.com> wrote:
> >
> > > +1
> > >
> > > The steps I've done:
> > >
> > > * Downloaded release tar and source tar, verified MD5.
> > > * Run a HDFS cluster, and copy files between local filesystem and HDFS.
> > >
> > >
> > > On Tue, Sep 27, 2016 at 1:28 PM, Sangjin Lee <sj...@apache.org> wrote:
> > > > Hi folks,
> > > >
> > > > I have created a release candidate RC0 for the Apache Hadoop 2.6.5
> > > release
> > > > (the next maintenance release in the 2.6.x release line). Below are
> the
> > > > details of this release candidate:
> > > >
> > > > The RC is available for validation at:
> > > > http://home.apache.org/~sjlee/hadoop-2.6.5-RC0/.
> > > >
> > > > The RC tag in git is release-2.6.5-RC0 and its git commit is
> > > > 6939fc935fba5651fdb33386d88aeb8e875cf27a.
> > > >
> > > > The maven artifacts are staged via repository.apache.org at:
> > > > https://repository.apache.org/content/repositories/
> > orgapachehadoop-1048/
> > > .
> > > >
> > > > You can find my public key at
> > > > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
> > > >
> > > > Please try the release and vote. The vote will run for the usual 5
> > days.
> > > > Huge thanks to Chris Trezzo for spearheading the release management
> and
> > > > doing all the work!
> > > >
> > > > Thanks,
> > > > Sangjin
> > >
> > >
> > >
> > > --
> > > Lei (Eddy) Xu
> > > Software Engineer, Cloudera
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

[VOTE] Release Apache Hadoop 2.6.5 (RC0)

[Sangjin Lee]

Hi folks,

I have created a release candidate RC0 for the Apache Hadoop 2.6.5 release
(the next maintenance release in the 2.6.x release line). Below are the
details of this release candidate:

The RC is available for validation at:
http://home.apache.org/~sjlee/hadoop-2.6.5-RC0/.

The RC tag in git is release-2.6.5-RC0 and its git commit is
6939fc935fba5651fdb33386d88aeb8e875cf27a.

The maven artifacts are staged via repository.apache.org at:
https://repository.apache.org/content/repositories/orgapachehadoop-1048/.

You can find my public key at
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.

Please try the release and vote. The vote will run for the usual 5 days.
Huge thanks to Chris Trezzo for spearheading the release management and
doing all the work!

Thanks,
Sangjin

[jira] [Resolved] (MAPREDUCE-6779) Mapreduce job failure on submission

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee resolved MAPREDUCE-6779.

Resolution: Resolved
  Assignee: Sangjin Lee

> Mapreduce job failure on submission
> ---
>
> Key: MAPREDUCE-6779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6779
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>        Assignee: Sangjin Lee
>Priority: Blocker
>
> Configure Hibench
> Try running Enhanced TestDFSIO
> {noformat}
> 2016-09-15 18:20:24,849 INFO mapreduce.Job: Task Id : 
> attempt_1473943118844_0001_m_01_0, Status : FAILED
> Error: java.lang.RuntimeException: Error in configuring object
> at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
> at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
> at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1806)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> ... 9 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2330)
> at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1108)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
> ... 14 more
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> Class org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2298)
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2322)
> ... 16 more
> Caused by: java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found
> at 
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2202)
> at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2296)
> ... 17 more
> {noformat}
> *mapreduce.JobResourceUploader: No job jar file set.  User classes may not be 
> found. See Job or Job#setJar(String).*
> Job jar is not getting set and not uploaded to staging dir



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [Release thread] 2.6.5 release activities

[Sangjin Lee]

We ported 16 issues to branch-2.6. We will go ahead and start the release
process, including cutting the release branch. If you have any critical
change that should be made part of 2.6.5, please reach out to us and commit
the changes. Thanks!

Sangjin

On Mon, Sep 12, 2016 at 3:24 PM, Sangjin Lee <sj...@apache.org> wrote:

> Thanks Chris!
>
> I'll help Chris to get those JIRAs marked in his spreadsheet committed.
> We'll cut the release branch shortly after that. If you have any critical
> change that should be made part of 2.6.5 (CVE patches included), please
> reach out to us and commit the changes. If all things go well, we'd like to
> cut the branch in a few days.
>
> Thanks,
> Sangjin
>
> On Fri, Sep 9, 2016 at 1:24 PM, Chris Trezzo <ctre...@gmail.com> wrote:
>
>> Hi all,
>>
>> I wanted to give an update on the Hadoop 2.6.5 release efforts.
>>
>> Here is what has been done so far:
>>
>> 1. I have gone through all of the potential backports and recorded the
>> commit hashes for each of them from the branch that seems the most
>> appropriate (i.e. if there was a backport to 2.7.x then I used the hash
>> from the backport).
>>
>> 2. I verified if the cherry pick for each commit is clean. This was best
>> effort as some of the patches are in parts of the code that I am less
>> familiar with. This is recorded in the public spread sheet here:
>> https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3ol
>> WpOCo6EBAey1WYC8hTRUemHvYPPzY/edit?usp=sharing
>>
>> I am going to need help from committers to get these backports committed.
>> If there are any committers that have some spare cycles, especially if you
>> were involved with the initial commit for one of these issues, please look
>> at the spreadsheet and volunteer to backport one of the issues.
>>
>> As always, please let me know if you have any questions or feel that I
>> have
>> missed something.
>>
>> Thank you!
>> Chris Trezzo
>>
>> On Mon, Aug 15, 2016 at 10:55 AM, Allen Wittenauer <
>> a...@effectivemachines.com
>> > wrote:
>>
>> >
>> > > On Aug 12, 2016, at 8:19 AM, Junping Du <j...@hortonworks.com> wrote:
>> > >
>> > >  In this community, we are so aggressive to drop Java 7 support in
>> 3.0.x
>> > release. Here, why we are so conservative to keep releasing new bits to
>> > support Java 6?
>> >
>> > I don't view a group of people putting bug fixes into a micro
>> > release as particularly conservative.  If a group within the community
>> > wasn't interested in doing it, 2.6.5 wouldn't be happening.
>> >
>> > But let's put the releases into context, because I think it
>> tells
>> > a more interesting story.
>> >
>> > * hadoop 2.6.x = EOLed JREs (6,7)
>> > * hadoop 2.7 -> hadoop 2.x = transitional (7,8)
>> > * hadoop 3.x = JRE 8
>> > * hadoop 4.x = JRE 9
>> >
>> > There are groups of people still using JDK6 and they want bug
>> > fixes in a maintenance release.  Boom, there's 2.6.x.
>> >
>> > Hadoop 3.x has been pushed off for years for "reasons".  So we
>> > still have releases coming off of branch-2.  If 2.7 had been released as
>> > 3.x, this chart would look less weird. But it wasn't thus 2.x has this
>> > weird wart in the middle of that supports JDK7 and JDK8.  Given the
>> public
>> > policy and roadmaps of at least one major vendor at the time of this
>> > writing, we should expect to see JDK7 support for at least the next two
>> > years after 3.x appears. Bang, there's 2.x, where x is some large
>> number.
>> >
>> > Then there is the future.  People using JRE 8 want to use newer
>> > dependencies.  A reasonable request. Some of these dependency updates
>> won't
>> > work with JRE 7.   We can't do that in hadoop 2.x in any sort of
>> compatible
>> > way without breaking the universe. (Tons of JIRAs on this point.) This
>> > means we can only do it in 3.x (re: Hadoop Compatibility Guidelines).
>> > Kapow, there's 3.x
>> >
>> > The log4j community has stated that v1 won't work with JDK9. In
>> > turn, this means we'll need to upgrade to v2 at some point.  Upgrading
>> to
>> > v2 will break the log4j properties file (and maybe other things?).
>> Another
>> > incompatible change and it likely won't appear until Apache Hadoop v4
>> > unless someone takes the initiative to fix it before v3 hits store
>> > shelves.  This makes JDK9 the likely target for Apache Hadoop v4.
>> >
>> > Having major release cadences tied to JRE updates isn't
>> > necessarily a bad thing and definitely forces the community to a)
>> actually
>> > stop beating around the bush on majors and b) actually makes it
>> relatively
>> > easy to determine what the schedule looks like to some degree.
>> >
>> >
>> >
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> >
>> >
>>
>
>

Re: [Release thread] 2.6.5 release activities

[Sangjin Lee]

Thanks Chris!

I'll help Chris to get those JIRAs marked in his spreadsheet committed.
We'll cut the release branch shortly after that. If you have any critical
change that should be made part of 2.6.5 (CVE patches included), please
reach out to us and commit the changes. If all things go well, we'd like to
cut the branch in a few days.

Thanks,
Sangjin

On Fri, Sep 9, 2016 at 1:24 PM, Chris Trezzo  wrote:

> Hi all,
>
> I wanted to give an update on the Hadoop 2.6.5 release efforts.
>
> Here is what has been done so far:
>
> 1. I have gone through all of the potential backports and recorded the
> commit hashes for each of them from the branch that seems the most
> appropriate (i.e. if there was a backport to 2.7.x then I used the hash
> from the backport).
>
> 2. I verified if the cherry pick for each commit is clean. This was best
> effort as some of the patches are in parts of the code that I am less
> familiar with. This is recorded in the public spread sheet here:
> https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3ol
> WpOCo6EBAey1WYC8hTRUemHvYPPzY/edit?usp=sharing
>
> I am going to need help from committers to get these backports committed.
> If there are any committers that have some spare cycles, especially if you
> were involved with the initial commit for one of these issues, please look
> at the spreadsheet and volunteer to backport one of the issues.
>
> As always, please let me know if you have any questions or feel that I have
> missed something.
>
> Thank you!
> Chris Trezzo
>
> On Mon, Aug 15, 2016 at 10:55 AM, Allen Wittenauer <
> a...@effectivemachines.com
> > wrote:
>
> >
> > > On Aug 12, 2016, at 8:19 AM, Junping Du  wrote:
> > >
> > >  In this community, we are so aggressive to drop Java 7 support in
> 3.0.x
> > release. Here, why we are so conservative to keep releasing new bits to
> > support Java 6?
> >
> > I don't view a group of people putting bug fixes into a micro
> > release as particularly conservative.  If a group within the community
> > wasn't interested in doing it, 2.6.5 wouldn't be happening.
> >
> > But let's put the releases into context, because I think it tells
> > a more interesting story.
> >
> > * hadoop 2.6.x = EOLed JREs (6,7)
> > * hadoop 2.7 -> hadoop 2.x = transitional (7,8)
> > * hadoop 3.x = JRE 8
> > * hadoop 4.x = JRE 9
> >
> > There are groups of people still using JDK6 and they want bug
> > fixes in a maintenance release.  Boom, there's 2.6.x.
> >
> > Hadoop 3.x has been pushed off for years for "reasons".  So we
> > still have releases coming off of branch-2.  If 2.7 had been released as
> > 3.x, this chart would look less weird. But it wasn't thus 2.x has this
> > weird wart in the middle of that supports JDK7 and JDK8.  Given the
> public
> > policy and roadmaps of at least one major vendor at the time of this
> > writing, we should expect to see JDK7 support for at least the next two
> > years after 3.x appears. Bang, there's 2.x, where x is some large number.
> >
> > Then there is the future.  People using JRE 8 want to use newer
> > dependencies.  A reasonable request. Some of these dependency updates
> won't
> > work with JRE 7.   We can't do that in hadoop 2.x in any sort of
> compatible
> > way without breaking the universe. (Tons of JIRAs on this point.) This
> > means we can only do it in 3.x (re: Hadoop Compatibility Guidelines).
> > Kapow, there's 3.x
> >
> > The log4j community has stated that v1 won't work with JDK9. In
> > turn, this means we'll need to upgrade to v2 at some point.  Upgrading to
> > v2 will break the log4j properties file (and maybe other things?).
> Another
> > incompatible change and it likely won't appear until Apache Hadoop v4
> > unless someone takes the initiative to fix it before v3 hits store
> > shelves.  This makes JDK9 the likely target for Apache Hadoop v4.
> >
> > Having major release cadences tied to JRE updates isn't
> > necessarily a bad thing and definitely forces the community to a)
> actually
> > stop beating around the bush on majors and b) actually makes it
> relatively
> > easy to determine what the schedule looks like to some degree.
> >
> >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
>

Re: [DISCUSS] Release cadence and EOL

[Sangjin Lee]

Thanks Karthik for opening a long overdue discussion on the release cadence
and EOL.

As for the EOL, I think we need to weigh between the benefit for the users
and the maintenance cost for the community. I'd also love to find out what
other (major) open source projects do in terms of the EOL.

Here is just an idea to get started. How about "a minor release line is
EOLed 2 years after it is released or there are 2 newer minor releases,
whichever is sooner. The community reserves the right to extend or shorten
the life of a release line if there is a good reason to do so."

The idea is to cap the maintenance at 2 years first, but also to consider
the actual alternatives. If there were 2 more minor releases, I think they
should be good alternatives for users to upgrade. That would also cap the
number of simultaneous maintenance lines at 2. I purposefully didn't
include major releases (e.g. 3.0.0) in this as it would take a much longer
time for users to upgrade from a previous major release.

Finally, I think it'd be good to have an escape clause for this so that the
community can make a collective decision to extend certain release lines if
it is deemed better for the community.

This is just a starting point for discussion. Thoughts?

Thanks,
Sangjin

On Fri, Aug 12, 2016 at 4:45 PM, Karthik Kambatla 
wrote:

> Forking off this discussion from 2.6.5 release thread. Junping and Chris T
> have brought up important concerns regarding too many concurrent releases
> and the lack of EOL for our releases.
>
> First up, it would be nice to hear from others on our releases having the
> notion of EOL and other predictability is indeed of interest.
>
> Secondly, I believe EOLs work better in conjunction with a predictable
> cadence. Given past discussions on this and current periods between our
> minor releases, I would like to propose a minor release on the latest major
> line every 6 months and a maintenance release on the latest minor release
> every 2 months.
>
> Eager to hear others thoughts.
>
> Thanks
> Karthik
>

Re: [Release thread] 2.6.5 release activities

[Sangjin Lee]

Thanks folks for opening up the discussion on our EOL policy. That's also
exactly what I wanted to discuss when I opened up the 2.6.5 discussion:

I also want to gauge the community's interest in maintaining the 2.6.x
> line. How long do we maintain this line? What would be a sensible EOL
> policy? Note that as the main code lines start diverging (java version,
> features, etc.), the cost of maintaining multiple release lines does go up.
> I'd love to hear your thoughts.


I look forward to that discussion thread. As for 2.6.5, since there is
enough interest in it, I'll work with Chris on that.

Regards,
Sangjin

On Fri, Aug 12, 2016 at 8:19 AM, Junping Du  wrote:

> I am OK with going forward for 2.6.5 release given some people may not be
> ready for migration to 2.7 and we have done some preparation work already.
> Thanks Chris for pushing the release work forward!
> About future releases of 2.6 (after 2.6.5) or any other legacy releases, I
> think it worth more discussions. I disagree with what Allen said below -
> Java 6 support should be a solid reason for branch-2.6 release keep going.
> In this community, we are so aggressive to drop Java 7 support in 3.0.x
> release. Here, why we are so conservative to keep releasing new bits to
> support Java 6? Our release effort, although driven by different person,
> should be consistent logically.
> I don't think we have clear rules/polices about EOL of legacy releases
> before. This is a bit off topic here. Will start a new thread later for
> more discussion.
>
> Thanks,
>
> Junping
>
> 
> From: Chris Trezzo 
> Sent: Friday, August 12, 2016 12:42 AM
> To: Karthik Kambatla
> Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: [Release thread] 2.6.5 release activities
>
> Thanks everyone for the discussion! Based on what has been said in this
> thread, I will move forward on the preparation efforts (working with
> Sangjin and the community) for a 2.6.5 release. If there is a strong
> objection to this, please let us know.
>
> I see three initial tasks going forward:
>
> 1. Fix the pre-commit build for branch-2.6. I am not entirely sure what is
> involved here, but will start looking into it using HADOOP-12800 as a
> starting point.
>
> 2. Look through the unresolved issues still targeted to 2.6.5 and
> resolve/re-target as necessary. There are currently 17 of them:
> https://issues.apache.org/jira/issues/?jql=(project%20%
> 3D%20%22Hadoop%20Common%22%20OR%20project%20%3D%20%
> 22Hadoop%20YARN%22%20OR%
> 20project%20%3D%20%22Hadoop%20Map%2FReduce%22%20OR%
> 20project%20%3D%20%22Hadoop%20HDFS%22)%20AND%20(
> fixVersion%20%3D%202.6.5%20OR%20%22Target%20Version%2Fs%22%
> 20%3D%202.6.5)%20AND%20(status%20!%3D%20Resolved%20AND%20status%20!%3D%
> 20Closed)%20ORDER%20BY%20status%20ASC
>
> 3. Look through the backport candidates in the spreadsheet in more detail
> and backport/drop as necessary. There are currently 15 of them:
> https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC
> 8hTRUemHvYPPzY/edit?usp=sharing
>
> Finally, I think it would be awesome to continue the release end-of-life
> discussion. As we get better and better at release cadence it is going to
> be really helpful to have an established EOL policy. It will be less
> confusing for contributors and help set clear expectations for end users as
> to when they need to start making reasonable migration plans, especially if
> they want to stay on a well supported release line. I would be happy to
> help with this effort as well. It would be great if we could leverage that
> discussion and have EOL information for the 2.6 line when we release 2.6.5.
>
> As always, please let me know if I have missed something.
>
> Thanks!
> Chris Trezzo
>
> On Thu, Aug 11, 2016 at 1:03 PM, Karthik Kambatla 
> wrote:
>
> > Since there is sufficient interest in 2.6.5, we should probably do it.
> All
> > the reasons Allen outlines make sense.
> >
> > That said, Junping brings up a very important point that we should think
> of
> > for future releases. For a new user or a user that does not directly
> > contribute to the project, more stable releases make it hard to pick
> from.
> >
> > As Chris T mentioned, the notion of EOL for our releases seems like a
> good
> > idea. However, to come up with any EOLs, we need to have some sort of
> > cadence for the releases. While this is hard for major releases (big
> bang,
> > potentially incompatible features), it should be doable for minor
> releases.
> >
> > How do people feel about doing a minor release every 6 months, with
> > follow-up maintenance releases every 2 months until the next minor and as
> > needed after that? That way, we could EOL a minor release a year after
> its
> > initial release? In the future, we could consider shrinking this window.
> In
> > addition to the 

Re: HADOOP-13410

[Sangjin Lee]

I just tested a simple service loader with the patch and it works fine even
if the jar is not in the classpath.

On Tue, Aug 9, 2016 at 10:41 AM, Sangjin Lee <sj...@apache.org> wrote:

> It uses ClassLoader.getResources() so there shouldn't be anything specific
> to the form of the resource (jar or not). But I'll test it later.
>
> On Tue, Aug 9, 2016 at 9:06 AM, Sean Busbey <bus...@cloudera.com> wrote:
>
>> ServiceLoader API stuff won't load out of the unpacked version, right?
>>
>> On Tue, Aug 9, 2016 at 11:00 AM, Sangjin Lee <sj...@apache.org> wrote:
>> > I'd like to get feedback from the community (especially those who might
>> > remember this) on HADOOP-13410:
>> > https://issues.apache.org/jira/browse/HADOOP-13410
>> >
>> > It appears that Hadoop's RunJar adds the original jar to the app's
>> > classpath even though the unjarred contents of the jar are in the
>> > classpath. As long as the file is a jar (or its variant), this seems
>> > completely superfluous. My suspicion is that the line of code that adds
>> the
>> > jar to the classpath may have been left there by accident.
>> >
>> > Could anyone confirm this? Does anyone see an issue with removing the
>> jar
>> > from the classpath? I've tested the fix with a couple of simple apps,
>> and I
>> > didn't see a problem.
>> >
>> > Thanks,
>> > Sangjin
>>
>>
>>
>> --
>> busbey
>>
>
>

Re: HADOOP-13410

[Sangjin Lee]

It uses ClassLoader.getResources() so there shouldn't be anything specific
to the form of the resource (jar or not). But I'll test it later.

On Tue, Aug 9, 2016 at 9:06 AM, Sean Busbey <bus...@cloudera.com> wrote:

> ServiceLoader API stuff won't load out of the unpacked version, right?
>
> On Tue, Aug 9, 2016 at 11:00 AM, Sangjin Lee <sj...@apache.org> wrote:
> > I'd like to get feedback from the community (especially those who might
> > remember this) on HADOOP-13410:
> > https://issues.apache.org/jira/browse/HADOOP-13410
> >
> > It appears that Hadoop's RunJar adds the original jar to the app's
> > classpath even though the unjarred contents of the jar are in the
> > classpath. As long as the file is a jar (or its variant), this seems
> > completely superfluous. My suspicion is that the line of code that adds
> the
> > jar to the classpath may have been left there by accident.
> >
> > Could anyone confirm this? Does anyone see an issue with removing the jar
> > from the classpath? I've tested the fix with a couple of simple apps,
> and I
> > didn't see a problem.
> >
> > Thanks,
> > Sangjin
>
>
>
> --
> busbey
>

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Sangjin Lee]

+1 (binding)

- downloaded both source and binary tarballs and verified the signatures
- set up a pseudo-distributed cluster
- ran some simple mapreduce jobs
- checked the basic web UI

Sangjin

On Wed, Jul 27, 2016 at 12:57 PM, John Zhuge  wrote:

> +1 (non-binding)
>
> - Build source with Java 1.8.0_101 on Centos 7.2 with native
> - Build source with Java 1.7.0_79 on Mac
> - Verify license and notice using the shell script in HADOOP-13374
> - Deploy a pseudo cluster
> - Run basic dfs, distcp, ACL, webhdfs commands
> - Run MapReduce workcount and pi examples
> - Start balancer
>
> Thanks,
> John
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Wed, Jul 27, 2016 at 11:38 AM, Robert Kanter 
> wrote:
>
> > +1 (binding)
> >
> > - Downloaded binary tarball
> > - verified signatures
> > - setup pseudo cluster
> > - ran some of the example jobs, clicked around the UI a bit
> >
> >
> > - Robert
> >
> >
> > On Mon, Jul 25, 2016 at 3:28 PM, Jason Lowe  >
> > wrote:
> >
> > > +1 (binding)
> > > - Verified signatures and digests- Built from source with native
> support-
> > > Deployed a pseudo-distributed cluster- Ran some sample jobs
> > > Jason
> > >
> > >   From: Vinod Kumar Vavilapalli 
> > >  To: "common-...@hadoop.apache.org" ;
> > > hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; "
> > > mapreduce-dev@hadoop.apache.org" 
> > > Cc: Vinod Kumar Vavilapalli 
> > >  Sent: Friday, July 22, 2016 9:15 PM
> > >  Subject: [VOTE] Release Apache Hadoop 2.7.3 RC0
> > >
> > > Hi all,
> > >
> > > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> > >
> > > As discussed before, this is the next maintenance release to follow up
> > > 2.7.2.
> > >
> > > The RC is available for validation at:
> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
> > >
> > > The RC tag in git is: release-2.7.3-RC0
> > >
> > > The maven artifacts are available via repository.apache.org <
> > > http://repository.apache.org/> at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> > <
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> > >
> > >
> > > The release-notes are inside the tar-balls at location
> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> > > hosted this at
> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
> > > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> > for
> > > your quick perusal.
> > >
> > > As you may have noted, a very long fix-cycle for the License & Notice
> > > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop
> > release)
> > > to slip by quite a bit. This release's related discussion thread is
> > linked
> > > below: [1].
> > >
> > > Please try the release and vote; the vote will run for the usual 5
> days.
> > >
> > > Thanks,
> > > Vinod
> > >
> > > [1]: 2.7.3 release plan:
> > >
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
> > <
> > > http://markmail.org/thread/6yv2fyrs4jlepmmr>
> > >
> > >
> > >
> >
>

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

[Sangjin Lee]

On Fri, Jul 22, 2016 at 5:15 PM, Allen Wittenauer <a...@effectivemachines.com>
wrote:

>
> But if I don't use ApplicationClassLoader, my java app is basically
> screwed then, right?
>

If we start upgrading the libraries aggressively, then it would also mean
that the ApplicationClassLoader should be more of the default than the
other way around (i.e. opt-out rather than opt-in). If we're not willing to
go there, then we cannot be too aggressive in upgrading libraries.

I'm not sure what you mean by "my java app is basically screwed", but if
you meant whether your java app would be OK if hadoop upgraded libraries
aggressively and you don't use the ApplicationClassLoader, then yes.


>
> Also:  right now, the non-Linux and/or non-x86 platforms have to supply
> their own leveldbjni jar (or at least the C level library?) in order to
> make YARN even functional.  How is that going to work with the class path
> manipulation?
>

First, the native libraries are orthogonal to this. They're not governed by
the java classpath.

For those platforms where users/admins need to provide their own LevelDB
libraries, the only requirement would be to add them to the
share/hadoop/.../lib directory. I don't think we would ask end users of the
clusters to bring in their own LevelDB library as it would not be an
end-user concern. I assume the administrators of clusters (still users but
not end users) would add it to the clusters. The classpath isolation
doesn't really have an impact on that.


>
> > On Jul 22, 2016, at 9:57 AM, Sangjin Lee <sj...@apache.org> wrote:
> >
> > The work on HADOOP-13070 and the ApplicationClassLoader are generic and
> go beyond YARN. It can be used in any JVM that uses hadoop. The current use
> cases are MR containers, hadoop's RunJar (as in "hadoop jar"), and the YARN
> node manager auxiliary services. I'm not sure if that's what you were
> asking, but I hope it helps.
> >
> > Regards,
> > Sangjin
> >
> > On Fri, Jul 22, 2016 at 9:16 AM, Sean Busbey <bus...@cloudera.com>
> wrote:
> > My work on HADOOP-11804 *only* helps processes that sit outside of YARN.
> :)
> >
> > On Fri, Jul 22, 2016 at 10:48 AM, Allen Wittenauer
> > <a...@effectivemachines.com> wrote:
> > >
> > > Does any of this work actually help processes that sit outside of YARN?
> > >
> > >> On Jul 21, 2016, at 12:29 PM, Sean Busbey <bus...@cloudera.com>
> wrote:
> > >>
> > >> thanks for bringing this up! big +1 on upgrading dependencies for 3.0.
> > >>
> > >> I have an updated patch for HADOOP-11804 ready to post this week. I've
> > >> been updating HBase's master branch to try to make use of it, but
> > >> could use some other reviews.
> > >>
> > >> On Thu, Jul 21, 2016 at 4:30 AM, Tsuyoshi Ozawa <oz...@apache.org>
> wrote:
> > >>> Hi developers,
> > >>>
> > >>> I'd like to discuss how to make an advance towards dependency
> > >>> management in Apache Hadoop trunk code since there has been lots work
> > >>> about updating dependencies in parallel. Summarizing recent works and
> > >>> activities as follows:
> > >>>
> > >>> 0) Currently, we have merged minimum update dependencies for making
> > >>> Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
> > >>> 1) After that, some people suggest that we should update the other
> > >>> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
> > >>> 2) In parallel, Sangjin and Sean are working on classpath isolation:
> > >>> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
> > >>>
> > >>> Main problems we try to solve in the activities above is as follows:
> > >>>
> > >>> * 1) tries to solve dependency hell between user-level jar and
> > >>> system(Hadoop)-level jar.
> > >>> * 2) tries to solve updating old libraries.
> > >>>
> > >>> IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries
> > >>> to separate class loader between client-side dependencies and
> > >>> server-side dependencies in Hadoop, so we can the change policy of
> > >>> updating libraries after doing 2). We can also decide which libraries
> > >>> can be shaded after 2).
> > >>>
> > >>> Hence, IMHO, a straight way we should go to is doing 2 at first.
> > >>> After that, we can update both client-side

Re: Setting JIRA fix versions for 3.0.0 releases

[Sangjin Lee]

On Thu, Jul 21, 2016 at 3:58 PM, Andrew Wang 
wrote:

> Thanks for the input Vinod, inline:
>
>
> > Similarly the list of features we are enabling in this alpha would be
> good
> > - may be update the Roadmap wiki. Things like classpath-isolation which
> > were part of the original 3.x roadmap are still not done.
> >
> > I already updated the website release notes at HADOOP-13383. I can update
> the Roadmap wiki and break out what's new in alpha1 too, thanks for the
> reminder.
>
>
> > > * Community bandwidth isn't zero-sum. This particularly applies to
> > people working on features that are only present in trunk, like EC, shell
> > script rewrite, etc.
> >
> >
> > A bunch of us are going to be busy with finishing 2.8.0. It isn’t
> > zero-sum, but it predicates those of us involved with 2.8.0 from looking
> at
> > it, even though we are very interested in doing so.
> >
> > There's a plan for more 3.0.0 alphas, so there's still time to help out
> before things settle down for beta. If 2.8.0 is ready to go, it should
> happen before even alpha2.
>
> >
> > Obviously, I am not making the case that this issue won’t happen ever. In
> > fact, this already happened with the parallel 2.6.x and 2.7.x releases.
> And
> > we precisely avoided major confusion there by lining up 2.7.2 behind
> 2.6.3
> > etc.
> >
>
> Could you clarify how lining up releases differently avoids the fix version
> issue?
>
> What we've been doing is something like "one fix version per active release
> line". Since we've revived 3.x as a release line, it seems like a lot of
> JIRAs need 3.x fix versions now.
>
> As an aside, I honestly don't know how to interpret the fix version
> instructions on the HowToCommit wiki. They don't seem to match up with what
> we actually do in practice.
>

I am also not quite sure I understand the rationale of what's in the
HowToCommit wiki. Assuming the semantic versioning (http://semver.org) as
our baseline thinking, having concurrent release streams alone breaks the
principle. And that is *regardless of* how we line up individual releases
in time (2.6.4 v. 2.7.3). Semantic versioning means 2.6.z < 2.7.* where *
is any number. Therefore, the moment we have any new 2.6.z release after
2.7.0, the rule is broken and remains that way. Timing of subsequent
releases is somewhat irrelevant.

>From a practical standpoint, I would love to know whether a certain patch
has been backported to a specific version. Thus, I would love to see fix
version enumerating all the releases that the JIRA went into. Basically the
more disclosure, the better. That would also make it easier for us
committers to see the state of the porting and identify issues like being
ported to 2.6.x but not to 2.7.x. What do you think? Should we revise our
policy?


>
> Best,
> Andrew
>

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

[Sangjin Lee]

The work on HADOOP-13070 and the ApplicationClassLoader are generic and go
beyond YARN. It can be used in any JVM that uses hadoop. The current use
cases are MR containers, hadoop's RunJar (as in "hadoop jar"), and the YARN
node manager auxiliary services. I'm not sure if that's what you were
asking, but I hope it helps.

Regards,
Sangjin

On Fri, Jul 22, 2016 at 9:16 AM, Sean Busbey  wrote:

> My work on HADOOP-11804 *only* helps processes that sit outside of YARN. :)
>
> On Fri, Jul 22, 2016 at 10:48 AM, Allen Wittenauer
>  wrote:
> >
> > Does any of this work actually help processes that sit outside of YARN?
> >
> >> On Jul 21, 2016, at 12:29 PM, Sean Busbey  wrote:
> >>
> >> thanks for bringing this up! big +1 on upgrading dependencies for 3.0.
> >>
> >> I have an updated patch for HADOOP-11804 ready to post this week. I've
> >> been updating HBase's master branch to try to make use of it, but
> >> could use some other reviews.
> >>
> >> On Thu, Jul 21, 2016 at 4:30 AM, Tsuyoshi Ozawa 
> wrote:
> >>> Hi developers,
> >>>
> >>> I'd like to discuss how to make an advance towards dependency
> >>> management in Apache Hadoop trunk code since there has been lots work
> >>> about updating dependencies in parallel. Summarizing recent works and
> >>> activities as follows:
> >>>
> >>> 0) Currently, we have merged minimum update dependencies for making
> >>> Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
> >>> 1) After that, some people suggest that we should update the other
> >>> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
> >>> 2) In parallel, Sangjin and Sean are working on classpath isolation:
> >>> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
> >>>
> >>> Main problems we try to solve in the activities above is as follows:
> >>>
> >>> * 1) tries to solve dependency hell between user-level jar and
> >>> system(Hadoop)-level jar.
> >>> * 2) tries to solve updating old libraries.
> >>>
> >>> IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries
> >>> to separate class loader between client-side dependencies and
> >>> server-side dependencies in Hadoop, so we can the change policy of
> >>> updating libraries after doing 2). We can also decide which libraries
> >>> can be shaded after 2).
> >>>
> >>> Hence, IMHO, a straight way we should go to is doing 2 at first.
> >>> After that, we can update both client-side and server-side
> >>> dependencies based on new policy(maybe we should discuss what kind of
> >>> incompatibility is acceptable, and the others are not).
> >>>
> >>> Thoughts?
> >>>
> >>> Thanks,
> >>> - Tsuyoshi
> >>>
> >>> -
> >>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >>>
> >>
> >>
> >>
> >> --
> >> busbey
> >>
> >> -
> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>
> >
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
>
>
>
> --
> busbey
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

[Sangjin Lee]

Thanks Tsuyoshi for opening the discussion. One benefit of the
dependency/classpath isolation work is that it can open up a possibility of
having diverging dependencies in a safe manner so that upgrading libraries
may have less impact. I'll spend some more time on HADOOP-13070 to make
some progress. Help is welcome there! :)

That said, upgrading libraries can still go on in parallel. If the past
experience is any guide, the hadoop dependencies badly trail current user
dependencies. If anything, we would be reducing the occurrences of problems
or workarounds people put in by upgrading our dependencies.

On Thu, Jul 21, 2016 at 2:30 AM, Tsuyoshi Ozawa  wrote:

> Hi developers,
>
> I'd like to discuss how to make an advance towards dependency
> management in Apache Hadoop trunk code since there has been lots work
> about updating dependencies in parallel. Summarizing recent works and
> activities as follows:
>
> 0) Currently, we have merged minimum update dependencies for making
> Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
> 1) After that, some people suggest that we should update the other
> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
> 2) In parallel, Sangjin and Sean are working on classpath isolation:
> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
>
> Main problems we try to solve in the activities above is as follows:
>
> * 1) tries to solve dependency hell between user-level jar and
> system(Hadoop)-level jar.
> * 2) tries to solve updating old libraries.
>
> IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries
> to separate class loader between client-side dependencies and
> server-side dependencies in Hadoop, so we can the change policy of
> updating libraries after doing 2). We can also decide which libraries
> can be shaded after 2).
>
> Hence, IMHO, a straight way we should go to is doing 2 at first.
> After that, we can update both client-side and server-side
> dependencies based on new policy(maybe we should discuss what kind of
> incompatibility is acceptable, and the others are not).
>
> Thoughts?
>
> Thanks,
> - Tsuyoshi
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6732:
--

 Summary: mapreduce tasks for YARN Timeline Service v.2: alpha 2
 Key: MAPREDUCE-6732
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Sangjin Lee
Assignee: Sangjin Lee


This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline 
Service v.2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6731:
--

 Summary: 
TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail 
for concurrent tests
 Key: MAPREDUCE-6731
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0-alpha1
Reporter: Sangjin Lee
Assignee: Sangjin Lee


{{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses 
the default file-system storage directory, and is brittle against concurrent 
tests.

We should use a unique storage directory for the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6331) [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee resolved MAPREDUCE-6331.

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha1

It's been merged to trunk. Huge thanks to the contributors who worked on this 
feature ([~jrottinghuis], [~djp], [~gtCarrera9], [~Naganarasimha], 
[~varun_saxena], [~vinodkv], [~vrushalic], and [~zjshen]) and everyone who 
participated in reviews and feedback!

> [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)
> 
>
> Key: MAPREDUCE-6331
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6331
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Vinod Kumar Vavilapalli
>        Assignee: Sangjin Lee
> Fix For: 3.0.0-alpha1
>
>
> Tracking umbrella for all MR changes to make it work with Timeline Service 
> Nextgen - YARN-2928.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Increased use of feature branches

[Sangjin Lee]

Thanks for your thoughts Anu.

Regarding your question

> And then comes the question, once 3.0 becomes official, where do we
> check-in a change,  if that would break something? so this will lead us
> back to trunk being the unstable – 3.0 being the new “branch-2”.


Andrew mentioned in the original email

> Regarding "trunk-incompat", since we're still in the alpha stage for
> 3.0.0, there's no need for this branch yet. This aspect of Vinod's proposal
> was still under a bit of discussion; Chris Douglas though we should cut a
> branch-3 for the first 3.0.0 beta, which aligns with my original thinking.
> This point doesn't necessarily need to be resolved now though, since again
> we're still doing alphas.


and I agree with that sentiment. I think even if we have a "trunk-incompat"
branch to hold future incompatible changes, the situation will change
little from today. Instead of dealing with "trunk" (where incompatible
changes may appear) and "branch-3", we would be dealing with
"trunk-incompat" and "trunk". Names are largely mnemonics then.


On Fri, Jun 10, 2016 at 12:37 PM, Anu Engineer <aengin...@hortonworks.com>
wrote:

> I actively work on two branches (Diskbalancer and ozone) and I agree with
> most of what Sangjin said.
> There is an overhead in working with branches, there are both technical
> costs and administrative issues
> which discourages developers from using branches.
>
> I think the biggest issue with branch based development is that fact that
> other developers do not use a branch.
> If a small feature appears as a series of commits to “”datanode.java””,
> the branch based developer ends up rebasing
> and paying this price of rebasing many times. If everyone followed a model
> of branch + Pull request, other branches
> would not have to deal with continues rebasing to trunk commits. If we are
> moving to a branch based
> development, we should probably move to that model for most development to
> avoid this tax on people who
>  actually end up working in the branches.
>
> I do have a question in my mind though: What is being proposed is that we
> move active development to branches
> if the feature is small or incomplete, however keep the trunk open for
> check-ins. One of the biggest reason why we
> check-in into trunk and not to branch-2 is because it is a change that
> will break backward compatibility. So do we
> have an expectation of backward compatibility thru the 3.0-alpha series (I
> personally vote No, since 3.0 is experimental
> at this stage), but if we decide to support some sort of backward-compact
> then willy-nilly committing to trunk
> and still maintaining the expectation we can release Alphas from 3.0 does
> not look possible.
>
> And then comes the question, once 3.0 becomes official, where do we
> check-in a change,  if that would break something?
> so this will lead us back to trunk being the unstable – 3.0 being the new
> “branch-2”.
>
> One more point: If we are moving to use a branch always – then we are
> looking at a model similar to using a git + pull
> request model. If that is so would it make sense to modify the rules to
> make these branches easier to merge?
> Say for example, if all commits in a branch has followed review and
> checking policy – just like trunk and commits
> have been made only after a sign off from a committer, would it be
> possible to merge with a 3-day voting period
> instead of 7, or treat it just like today’s commit to trunk – but with 2
> people signing-off?
>
> What I am suggesting is reducing the administrative overheads of using a
> branch to encourage use of branching.
> Right now it feels like Apache’s process encourages committing directly to
> trunk than a branch
>
> Thanks
> Anu
>
>
> On 6/10/16, 10:50 AM, "sjl...@gmail.com on behalf of Sangjin Lee" <
> sjl...@gmail.com on behalf of sj...@apache.org> wrote:
>
> >Having worked on a major feature in a feature branch, I have some thoughts
> >and observations on feature branch development.
> >
> >IMO feature branch development v. direct commits to trunk in piecemeal is
> >really a choice of *granularity*. Do we want a series of fine-grained
> state
> >changes on trunk or fewer coarse-grained chunks of commits on trunk?
> >
> >This makes me favor a branch-based development model for any
> "decent-sized"
> >features (we'll need to define "decent-sized" of course). Once you have
> >coarse-grained changes, it's easier to reason about what made what release
> >and in what state. As importantly, it makes it easier to back out a
> >complete feature fairly easily if that becomes necessary. My totall

Re: [DISCUSS] Increased use of feature branches

[Sangjin Lee]

Having worked on a major feature in a feature branch, I have some thoughts
and observations on feature branch development.

IMO feature branch development v. direct commits to trunk in piecemeal is
really a choice of *granularity*. Do we want a series of fine-grained state
changes on trunk or fewer coarse-grained chunks of commits on trunk?

This makes me favor a branch-based development model for any "decent-sized"
features (we'll need to define "decent-sized" of course). Once you have
coarse-grained changes, it's easier to reason about what made what release
and in what state. As importantly, it makes it easier to back out a
complete feature fairly easily if that becomes necessary. My totally
unscientific suggestion may be if a feature takes more than dozen commits
and longer than a month, we should probably have a bias towards a feature
branch.

Branch-based development also makes you go faster if your feature is
larger. I wouldn't do it the other way for timeline service v.2 for example.

That said, feature branches don't come for free. Now the onus is on the
feature developer to constantly rebase with the trunk to keep it reasonably
integrated with the trunk. More logistics is involved for the feature
developer. Another big question is, when a feature branch gets big and it's
time to merge, would it get as scrutinized as a series of individual
commits? Since the size of merge can be big, you kind of have to rely on
those feature committers and those who help them.

In terms of integrating/stabilizing, I don't think branch development
necessarily makes it harder. It is again granularity. In case of direct
commits on trunk, you do a lot more fine-grained integrations. In case of
branch development, you do far fewer coarse-grained integrations via
rebasing. If more people are doing branch-based development, it makes
rebasing easier to manage too.

Going back to the related topic of where to release (trunk v. branch-X), I
think that is more of a proxy of the real question of "how do we maintain
quality and stability of the trunk?". Even if we release from the trunk, if
our bar for merging to trunk is low, the quality will not improve
automatically. So I think we ought to tackle the quality question first.

My 2 cents.


On Fri, Jun 10, 2016 at 8:57 AM, Zhe Zhang  wrote:

> Thanks for the notes Andrew, Junping, Karthik.
>
> Here are some of my understandings:
>
> - Trunk is the "latest and greatest" of Hadoop. If a user starts using
> Hadoop today, without legacy workloads, trunk is what he/she should use.
> - Therefore, each commit to trunk should be transactional -- atomic,
> consistent, isolated (from other uncommitted patches); I'm not so sure
> about durability, Hadoop might be gone in 50 years :). As a committer, I
> should be able to look at a patch and determine whether it's a
> self-contained improvement of trunk, without looking at other uncommitted
> patches.
> - Some comments inline:
>
> On Fri, Jun 10, 2016 at 6:56 AM Junping Du  wrote:
>
> > Comparing with advantages, I believe the disadvantages of shipping any
> > releases directly from trunk are more obvious and significant:
> > - A lot of commits (incompatible, risky, uncompleted feature, etc.) have
> > to wait to commit to trunk or put into a separated branch that could
> delay
> > feature development progress as additional vote process get involved even
> > the feature is simple and harmless.
> >
> Thanks Junping, those are valid concerns. I think we should clearly
> separate incompatible with  uncompleted / half-done work in this
> discussion. Whether people should commit incompatible changes to trunk is a
> much more tricky question (related to trunk-incompat etc.). But per my
> comment above, IMHO, *not committing uncompleted work to trunk* should be a
> much easier principle to agree upon.
>
>
> > - For small feature with only 1 or 2 commits, that need three +1 from
> PMCs
> > will increase the bar largely for contributors who just start to
> contribute
> > on Hadoop features but no such sufficient support.
> >
> Development overhead is another valid concern. I think our rule-of-thumb
> should be that, small-medium new features should be proposed as a single
> JIRA/patch (as we recently did for HADOOP-12666). If the complexity goes
> beyond a single JIRA/patch, use a feature branch.
>
>
> >
> > Given these concerns, I am open to other options, like: proposed by Vinod
> > or Chris, but rather than to release anything directly from trunk.
> >
> > - This point doesn't necessarily need to be resolved now though, since
> > again we're still doing alphas.
> > No. I think we have to settle down this first. Without a common agreed
> and
> > transparent release process and branches in community, any release
> (alpha,
> > beta) bits is only called a private release but not a official apache
> > hadoop release (even alpha).
> >
> >
> > Thanks,
> >
> > Junping
> > 
> 

Re: [Release thread] 2.8.0 release activities

[Sangjin Lee]

I see the following list of JIRAs from HADOOP and HDFS as blockers for
2.8.0. Other JIRAs are either old issues with little movement or new issues
that don't appear to be as critical/ready.

- HADOOP-12893
- HADOOP-12892
- HADOOP-10940 (patch ready?)
- HADOOP-12971 (can be done relatively quickly?)
- HDFS-7959
- HDFS-7597 (needs review/some more work?)

I would propose moving the rest out of scope for 2.8.0 (23 JIRAs). Let me
know what you think.


On Wed, May 11, 2016 at 5:37 PM, Wangda Tan <wheele...@gmail.com> wrote:

> Sounds good to me :).
>
> Jian and I have looked at all existing 2.8.0 blockers and criticals today.
> To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
> out. Left comments on these JIRAs asked original owners, plan to update
> target version of these JIRAs early next week.
>
> Will keep this thread updated.
>
> Thanks,
> Wangda
>
>
> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee <sj...@apache.org> wrote:
>
>> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
>> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
>> report the list here. Then folks can chime in if you agree
>>
>> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
>>
>> Thanks,
>> Sangjin
>>
>> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan <wheele...@gmail.com> wrote:
>>
>>> +1, we should close such staled JIRAs to avoid doing unnecessary checks
>>> for
>>> every releases.
>>>
>>> I'm working on reviewing YARN/MR critical/blocker patches currently, it
>>> gonna very helpful if someone else can help with reviewing Common/HDFS
>>> JIRAs.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee <sj...@apache.org> wrote:
>>>
>>> > Where do we stand in terms of closing out blocker/critical issues for
>>> > 2.8.0? I still see 50 open JIRAs in Vinod's list:
>>> > https://issues.apache.org/jira/issues/?filter=12334985
>>> >
>>> > But I see a lot of JIRAs with no patches or very stale patches. It
>>> would be
>>> > a good exercise to come up with the list of JIRAs that we need to block
>>> > 2.8.0 for and focus our attention on closing them out. Thoughts?
>>> >
>>> > Thanks,
>>> > Sangjin
>>> >
>>> > On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran <
>>> ste...@hortonworks.com>
>>> > wrote:
>>> >
>>> > >
>>> > > > On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
>>> vino...@apache.org>
>>> > > wrote:
>>> > > >
>>> > > > We are not converging - there’s still 58 more. I need help from the
>>> > > community in addressing / review 2.8.0 blockers. If folks can start
>>> with
>>> > > reviewing Patch available tickets, that’ll be great.
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > > I'm still doing the s3a stuff, other people testing and reviewing
>>> this
>>> > > stuff welcome.
>>> > >
>>> > > in particular, I could do with others playing with this patch of
>>> mine,
>>> > > which adds counters and things into S3a, based on the azure
>>> > instrumentation
>>> > >
>>> > > https://issues.apache.org/jira/browse/HADOOP-13028
>>> > >
>>> > >
>>> > >
>>> >
>>>
>>
>>
>

Re: [Release thread] 2.8.0 release activities

[Sangjin Lee]

How about this? I'll review the HADOOP/HDFS bugs in that list to come up
with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
report the list here. Then folks can chime in if you agree

Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?

Thanks,
Sangjin

On Wed, May 11, 2016 at 4:26 PM, Wangda Tan <wheele...@gmail.com> wrote:

> +1, we should close such staled JIRAs to avoid doing unnecessary checks for
> every releases.
>
> I'm working on reviewing YARN/MR critical/blocker patches currently, it
> gonna very helpful if someone else can help with reviewing Common/HDFS
> JIRAs.
>
> Thanks,
> Wangda
>
>
> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee <sj...@apache.org> wrote:
>
> > Where do we stand in terms of closing out blocker/critical issues for
> > 2.8.0? I still see 50 open JIRAs in Vinod's list:
> > https://issues.apache.org/jira/issues/?filter=12334985
> >
> > But I see a lot of JIRAs with no patches or very stale patches. It would
> be
> > a good exercise to come up with the list of JIRAs that we need to block
> > 2.8.0 for and focus our attention on closing them out. Thoughts?
> >
> > Thanks,
> > Sangjin
> >
> > On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran <ste...@hortonworks.com>
> > wrote:
> >
> > >
> > > > On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> > > wrote:
> > > >
> > > > We are not converging - there’s still 58 more. I need help from the
> > > community in addressing / review 2.8.0 blockers. If folks can start
> with
> > > reviewing Patch available tickets, that’ll be great.
> > > >
> > > >
> > >
> > >
> > > I'm still doing the s3a stuff, other people testing and reviewing this
> > > stuff welcome.
> > >
> > > in particular, I could do with others playing with this patch of mine,
> > > which adds counters and things into S3a, based on the azure
> > instrumentation
> > >
> > > https://issues.apache.org/jira/browse/HADOOP-13028
> > >
> > >
> > >
> >
>

Re: [Release thread] 2.8.0 release activities

[Sangjin Lee]

Where do we stand in terms of closing out blocker/critical issues for
2.8.0? I still see 50 open JIRAs in Vinod's list:
https://issues.apache.org/jira/issues/?filter=12334985

But I see a lot of JIRAs with no patches or very stale patches. It would be
a good exercise to come up with the list of JIRAs that we need to block
2.8.0 for and focus our attention on closing them out. Thoughts?

Thanks,
Sangjin

On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 
wrote:

>
> > On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli 
> wrote:
> >
> > We are not converging - there’s still 58 more. I need help from the
> community in addressing / review 2.8.0 blockers. If folks can start with
> reviewing Patch available tickets, that’ll be great.
> >
> >
>
>
> I'm still doing the s3a stuff, other people testing and reviewing this
> stuff welcome.
>
> in particular, I could do with others playing with this patch of mine,
> which adds counters and things into S3a, based on the azure instrumentation
>
> https://issues.apache.org/jira/browse/HADOOP-13028
>
>
>

[jira] [Resolved] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance

[Sangjin Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee resolved MAPREDUCE-6372.

   Resolution: Resolved
Fix Version/s: YARN-2928

I have addressed the known issues as part of MAPREDUCE-6546.

> clean up several issues with TimelineServicePerformance
> ---
>
> Key: MAPREDUCE-6372
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>        Reporter: Sangjin Lee
>    Assignee: Sangjin Lee
>  Labels: yarn-2928-1st-milestone
> Fix For: YARN-2928
>
>
> We found a few issues with the TimelineServicePerformanceV2 test driver while 
> running it for the performance tests. Filing this JIRA to fix those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Looking to a Hadoop 3 release

[Sangjin Lee]

Another thing to throw in there is the dependency/classpath isolation
(HADOOP-11656). Some efforts have already been made by Sean, and it'd be
great to complete this to have a much better dependency isolation solution
for 3.x.

On Thu, Feb 18, 2016 at 5:33 PM, Gangumalla, Uma 
wrote:

> Yes. I think starting 3.0 release with alpha is good idea. So it would get
> some time to reach the beta or GA.
>
> +1 for the plan.
>
> For the compatibility purposes and as current stable versions, we should
> continue 2.x releases anyway.
>
> Thanks Andrew for starting the thread.
>
> Regards,
> Uma
>
> On 2/18/16, 3:04 PM, "Andrew Wang"  wrote:
>
> >Hi Kihwal,
> >
> >I think there's still value in continuing the 2.x releases. 3.x comes with
> >the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
> >be beta or GA for some number of months. In the meanwhile, it'd be good to
> >keep putting out regular, stable 2.x releases.
> >
> >Best,
> >Andrew
> >
> >
> >On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee  >
> >wrote:
> >
> >> Moving Hadoop 3 forward sounds fine. If EC is one of the main
> >>motivations,
> >> are we getting rid of branch-2.8?
> >>
> >> Kihwal
> >>
> >>   From: Andrew Wang 
> >>  To: "common-...@hadoop.apache.org" 
> >> Cc: "yarn-...@hadoop.apache.org" ; "
> >> mapreduce-dev@hadoop.apache.org" ;
> >> hdfs-dev 
> >>  Sent: Thursday, February 18, 2016 4:35 PM
> >>  Subject: Re: Looking to a Hadoop 3 release
> >>
> >> Hi all,
> >>
> >> Reviving this thread. I've seen renewed interest in a trunk release
> >>since
> >> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
> >>the
> >> shell script rewrite, and many other improvements, I think it's time to
> >> revisit Hadoop 3.0 release plans.
> >>
> >> My overall plan is still the same as in my original email: a series of
> >> regular alpha releases leading up to beta and GA. Alpha releases make it
> >> easier for downstreams to integrate with our code, and making them
> >>regular
> >> means features can be included when they are ready.
> >>
> >> I know there are some incompatible changes waiting in the wings
> >> (i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
> >> HADOOP-9991 bumping dependency versions) that would be good to get in.
> >>If
> >> you have changes like this, please set the target version to 3.0.0 and
> >>mark
> >> them "Incompatible". We can use this JIRA query to track:
> >>
> >>
> >>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HD
> >>FS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%
> >>223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flag
> >>s%22%3D%22Incompatible%20change%22%20order%20by%20priority
> >>
> >> There's some release-related stuff that needs to be sorted out (namely,
> >>the
> >> new CHANGES.txt and release note generation from Yetus), but I'd
> >> tentatively like to roll the first alpha a month out, so third week of
> >> March.
> >>
> >> Best,
> >> Andrew
> >>
> >> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata 
> >>wrote:
> >>
> >> > Avoiding the use of JDK8 language features (and, presumably, APIs)
> >> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> >> > source version to JDK8.
> >> >
> >> > Also, note that releasing from trunk is a way of achieving #3, it's
> >> > not a way of abandoning it.
> >> >
> >> >
> >> >
> >> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang  >
> >> > wrote:
> >> > > Hi Raymie,
> >> > >
> >> > > Konst proposed just releasing off of trunk rather than cutting a
> >> > branch-2,
> >> > > and there was general agreement there. So, consider #3 abandoned.
> >>1&2
> >> can
> >> > > be achieved at the same time, we just need to avoid using JDK8
> >>language
> >> > > features in trunk so things can be backported.
> >> > >
> >> > > Best,
> >> > > Andrew
> >> > >
> >> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata 
> >> > wrote:
> >> > >
> >> > >> In this (and the related threads), I see the following three
> >> > requirements:
> >> > >>
> >> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> >> > >>
> >> > >> 2. "We'll still be releasing 2.x releases for a while, with similar
> >> > >> feature sets as 3.x."
> >> > >>
> >> > >> 3. Avoid the "risk of split-brain behavior" by "minimize
> >>backporting
> >> > >> headaches. Pulling trunk > branch-2 > branch-2.x is already
> >>tedious.
> >> > >> Adding a branch-3, branch-3.x would be obnoxious."
> >> > >>
> >> > >> These three cannot be achieved at the same time.  Which do we
> >>abandon?
> >> > >>
> >> > >>
> >> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
> >>
> >> 

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

[Sangjin Lee]

+1 (not binding)

- downloaded both the source and the binary tarballs, and checked the
checksums and signatures
- started a pseudo-distributed cluster and ran test MR jobs
- spot checked the URI and the logs

Sangjin

On Mon, Feb 8, 2016 at 4:17 PM, Jason Lowe 
wrote:

> +1 (binding)
> - verified signatures and digests- built native from source- deployed a
> single-node cluster and ran some sample MapReduce jobs.
> Jason
>
>
>   From: Junping Du 
>  To: "hdfs-...@hadoop.apache.org" ; "
> yarn-...@hadoop.apache.org" ; "
> mapreduce-dev@hadoop.apache.org" ; "
> common-...@hadoop.apache.org" 
>  Sent: Wednesday, February 3, 2016 1:01 AM
>  Subject: [VOTE] Release Apache Hadoop 2.6.4 RC0
>
> Hi community folks,
>   I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
> maintenance release to follow up 2.6.3.) according to email thread of
> release plan 2.6.4 [1]. Below is details of this release candidate:
>
> The RC is available for validation at:
> *http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
> *
>
> The RC tag in git is: release-2.6.4-RC0
>
> The maven artifacts are staged via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
>  >*
>
> You can find my public key at:
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> Please try the release and vote. The vote will run for the usual 5 days.
>
> Thanks!
>
>
> Cheers,
>
> Junping
>
>
> [1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
>
>
>
>

Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale branches

[Sangjin Lee]

I deleted all the obsolete 2928 branches (except for YARN-2928 and
feature-YARN-2928) a few days ago. Do you still see them?

On Fri, Jan 15, 2016 at 5:54 AM, Junping Du <j...@hortonworks.com> wrote:

> There are also several stale branches for YARN-2928 that we should
> consolidate:
>   remotes/origin/YARN-2928
>   remotes/origin/YARN-2928-new
>   remotes/origin/YARN-2928-old
>   remotes/origin/YARN-2928-old-2015-11-09
>   remotes/origin/YARN-2928-rebase
>   remotes/origin/feature-YARN-2928  (the latest one)
> I think we should remove all stale ones and rename the latest one
> feature-YARN-2928 to YARN-2928 to follow our practice for branch
> development. Thoughts?
>
>
> Thanks,
>
> Junping
> 
> From: sjl...@gmail.com <sjl...@gmail.com> on behalf of Sangjin Lee <
> sj...@apache.org>
> Sent: Friday, January 15, 2016 7:17 AM
> To: yarn-...@hadoop.apache.org
> Cc: Hadoop Common; hdfs-...@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; Vinod Kumar Vavilapalli
> Subject: Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale
> branches
>
> Thanks Vinod for the proposal. I deleted my branch (sjlee/hdfs-merge) the
> other day.
>
> Sangjin
>
> On Thu, Jan 14, 2016 at 1:26 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org
> > wrote:
>
> > Hi all,
> >
> > As some of you have noticed, we have an update from ASF infra on git
> > branching policy: We no longer have a ASF wide mandate on disallowing
> > force-pushes on all branches / tags.
> >
> > Summarizing information from the INFRA email for the sake of clarity in
> > the midst of recent confusion
> >  - We now can do force pushes, and branch/tag deletion on any branch or
> > tag except refs/tags/rel
> >  - Any force pushes will be annotated in the commit-email as “[Forced
> > Update!]” for the community to watch out for undesired force-pushes
> >  - Only tags under refs/tags/rel are protected from force-push for the
> > sake of release-provenance: Essentially, the releases that community
> votes
> > on are archived in their entirety with the development history and we
> > cannot alter that once a tag is created. As one might expect.
> >
> > What this means for us
> >  - Stale branches: There are a few stale branches that got accumulated.
> > — During this branch moratorium, origin/bracnh-2.8 got created (May
> be
> > as part of HDFS-8785, can’t say for sure)
> > — A couple of stale branches that helped 2.6.1 release:
> > origin/sjlee/hdfs-merge and origin/ajisakaa/common-merge
> > — I’ll wait till EOD tomorrow for any yays/nays and delete them
> >  - Feature branch updates: Developers can now go rebase and force-push
> > their feature branches.
> >  - Mainline branches: Mainline branches like trunk, branch-2 have always
> > been configured to avoid force-pushes. In general, force-push continues
> to
> > be recommended mainly for feature branches and definitely not on any
> > mainline branches from which we make releases.
> >  - Release tags:
> > — To follow ASF provenance policy, we will now push the final release
> > tags under refs/tags/rel. We will first push the RC tags under where they
> > reside now (refs/tags) and if the vote passes, the final tag will be
> > created under refs/tags/rel.
> > — I’ll update our release wiki page
> > http://wiki.apache.org/hadoop/HowToRelease <
> > http://wiki.apache.org/hadoop/HowToRelease> with the same details once I
> > can get 2.7.2 release done using this updated process.
> >  - Existing release tags:
> > — There is a general recommendation from INFRA team to take all of
> our
> > existing release tags under "tags" and copy them to “rel”.
> > — I’ll wait till EOD tomorrow for any yays/nays and copy existing
> > releases under refs/tags/rel following general recommendations.
> >
> > Any comments / thoughts / questions welcome.
> >
> > Thanks
> > +Vinod
>

Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale branches

[Sangjin Lee]

Thanks Vinod for the proposal. I deleted my branch (sjlee/hdfs-merge) the
other day.

Sangjin

On Thu, Jan 14, 2016 at 1:26 PM, Vinod Kumar Vavilapalli  wrote:

> Hi all,
>
> As some of you have noticed, we have an update from ASF infra on git
> branching policy: We no longer have a ASF wide mandate on disallowing
> force-pushes on all branches / tags.
>
> Summarizing information from the INFRA email for the sake of clarity in
> the midst of recent confusion
>  - We now can do force pushes, and branch/tag deletion on any branch or
> tag except refs/tags/rel
>  - Any force pushes will be annotated in the commit-email as “[Forced
> Update!]” for the community to watch out for undesired force-pushes
>  - Only tags under refs/tags/rel are protected from force-push for the
> sake of release-provenance: Essentially, the releases that community votes
> on are archived in their entirety with the development history and we
> cannot alter that once a tag is created. As one might expect.
>
> What this means for us
>  - Stale branches: There are a few stale branches that got accumulated.
> — During this branch moratorium, origin/bracnh-2.8 got created (May be
> as part of HDFS-8785, can’t say for sure)
> — A couple of stale branches that helped 2.6.1 release:
> origin/sjlee/hdfs-merge and origin/ajisakaa/common-merge
> — I’ll wait till EOD tomorrow for any yays/nays and delete them
>  - Feature branch updates: Developers can now go rebase and force-push
> their feature branches.
>  - Mainline branches: Mainline branches like trunk, branch-2 have always
> been configured to avoid force-pushes. In general, force-push continues to
> be recommended mainly for feature branches and definitely not on any
> mainline branches from which we make releases.
>  - Release tags:
> — To follow ASF provenance policy, we will now push the final release
> tags under refs/tags/rel. We will first push the RC tags under where they
> reside now (refs/tags) and if the vote passes, the final tag will be
> created under refs/tags/rel.
> — I’ll update our release wiki page
> http://wiki.apache.org/hadoop/HowToRelease <
> http://wiki.apache.org/hadoop/HowToRelease> with the same details once I
> can get 2.7.2 release done using this updated process.
>  - Existing release tags:
> — There is a general recommendation from INFRA team to take all of our
> existing release tags under "tags" and copy them to “rel”.
> — I’ll wait till EOD tomorrow for any yays/nays and copy existing
> releases under refs/tags/rel following general recommendations.
>
> Any comments / thoughts / questions welcome.
>
> Thanks
> +Vinod

[jira] [Created] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6577:
--

 Summary: MR AM unable to load native library without 
MR_AM_ADMIN_USER_ENV set
 Key: MAPREDUCE-6577
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.6.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Priority: Critical


If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not 
configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:

{noformat}
2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
Unable to load native-hadoop library for your platform... using builtin-java 
classes where applicable
{noformat}

As a result, any code that needs the hadoop native library in the MR AM will 
fail. For example, an uber-AM with lz4 compression for the mapper task will 
fail:
{noformat}
2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] 
org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
(uberized) 'child' : java.lang.RuntimeException: native lz4 library not 
available
at 
org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
at 
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at 
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

[Sangjin Lee]

+1 (non-binding)

- downloaded source and binary and verified the signatures (although I
didn't connect with Junping via web of trust)
- started a pseudo-distributed cluster and ran test jobs
- browsed the RM and NN UI
- looked through the daemon logs

Thanks Junping.

Sangjin

On Wed, Dec 16, 2015 at 5:11 AM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> +1 (non-binding)
>
> -- Downloaded both source and binary tarballs successfully.
> --Set up a pseudo-distributed cluster and Distributed HA Cluster
> --Ran Several jobs Slive,Terasort and pi.
> --All are working fine.
>
>
> Thanks & Regards
>  Brahma Reddy Battula
> 
> From: Steve Loughran [ste...@hortonworks.com]
> Sent: Wednesday, December 16, 2015 5:58 PM
> To: mapreduce-dev@hadoop.apache.org
> Cc: Hadoop Common; hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org;
> junping...@apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.6.3 RC0
>
> +1, binding
>
>
> Did a build and test with slider with the build set to
> -Dhadoop.version=2.6.3
>
> this does a D/L and test from the staging repo. All artifacts were
> located, and the tests completed
>
> > On 12 Dec 2015, at 00:16, Junping Du  wrote:
> >
> >
> > Hi all developers in hadoop community,
> >   I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the next
> maintenance release to follow up 2.6.2.) according to email thread of
> release plan 2.6.3 [1]. Sorry for this RC coming a bit late as several
> blocker issues were getting committed until yesterday. Below is the details:
> >
> > The RC is available for validation at:
> > *http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
> > *
> >
> > The RC tag in git is: release-2.6.3-RC0
> >
> > The maven artifacts are staged via repository.apache.org at:
> > *
> https://repository.apache.org/content/repositories/orgapachehadoop-1025/?
> > <
> https://repository.apache.org/content/repositories/orgapachehadoop-1025/>*
> >
> > You can find my public key at:
> > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
> >
> > Please try the release and vote. The vote will run for the usual 5 days.
> >
> > Thanks and happy weekend!
> >
> >
> > Cheers,
> >
> > Junping
> >
> >
> > [1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y
> >
>
>

Re: continuing releases on Apache Hadoop 2.6.x

[Sangjin Lee]

It would be great if we can get enough number of fixes by early December.
18 seems bit on the low side, but if we lose this window it won't be until
next year.

As for the release management, thanks Chris, Junping, and Haohui for
volunteering! I'll reach out to you to discuss what we do with 2.6.3. I
assume we will have more maintenance releases in the 2.6.x line, so there
will be more opportunities. We do need one person with PMC privileges to be
able to go through all the release management steps without assistance,
which I learned last time.

Regards,
Sangjin

On Fri, Nov 20, 2015 at 10:03 AM, Sean Busbey <bus...@cloudera.com> wrote:

> Early december would be great, presuming the RC process doesn't take too
> long. By then it'll already have over a month since the 2.6.2 release and
> I'm sure the folks contributing the 18 patches we already have in would
> like to see their work out there.
>
> On Fri, Nov 20, 2015 at 7:51 AM, Junping Du <j...@hortonworks.com> wrote:
>
> > +1. Early Dec sounds too early for 2.6.3 release given we only have 18
> > patches since recently release 2.6.2.
> > We should nominate more fixes and wait a while for the feedback on 2.6.2.
> >
> > Thanks,
> >
> > Junping
> > 
> > From: Vinod Vavilapalli <vino...@hortonworks.com>
> > Sent: Thursday, November 19, 2015 11:34 PM
> > To: yarn-...@hadoop.apache.org
> > Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > mapreduce-dev@hadoop.apache.org
> > Subject: Re: continuing releases on Apache Hadoop 2.6.x
> >
> > I see 18 JIRAs across the sub-projects as of now in 2.6.3. Seems like we
> > will have a reasonable number of fixes if we start an RC early december.
> >
> > In the mean while, we should also review 2.7.3 and 2.8.0 blocker /
> > critical list and see if it makes sense to backport any of those into
> 2.6.3.
> >
> > +Vinod
> >
> >
> > On Nov 17, 2015, at 5:10 PM, Sangjin Lee <sj...@apache.org > sj...@apache.org>> wrote:
> >
> > I'd like to pick up this email discussion again. It is time that we
> started
> > thinking about the next release in the 2.6.x line. IMO we want to walk
> the
> > balance between maintaining a reasonable release cadence and getting a
> good
> > amount of high-quality fixes. The timeframe is a little tricky as the
> > holidays are approaching. If we have enough fixes accumulated in
> > branch-2.6, some time early December might be a good target for cutting
> the
> > first release candidate. Once we miss that window, I think we are looking
> > at next January. I'd like to hear your thoughts on this.
> >
> >
>
>
> --
> Sean
>

Re: continuing releases on Apache Hadoop 2.6.x

[Sangjin Lee]

I'd like to pick up this email discussion again. It is time that we started
thinking about the next release in the 2.6.x line. IMO we want to walk the
balance between maintaining a reasonable release cadence and getting a good
amount of high-quality fixes. The timeframe is a little tricky as the
holidays are approaching. If we have enough fixes accumulated in
branch-2.6, some time early December might be a good target for cutting the
first release candidate. Once we miss that window, I think we are looking
at next January. I'd like to hear your thoughts on this.

It'd be good if someone can volunteer for the release manager for 2.6.3.
I'd be happy to help out in any way I can. Thanks!

Regards,
Sangjin

On Mon, Nov 2, 2015 at 11:45 AM, Vinod Vavilapalli <vino...@hortonworks.com>
wrote:

> Just to stress on the following, it is very important that any critical
> bug-fixes that we push into 2.8.0 or even trunk, we should consider them
> for 2.6.3 and 2.7.3 if it makes sense. This is the only way we can avoid
> extremely long release cycles like that of 2.6.1.
>
> Also, to clarify a little, use Target-version if you want a discussion of
> the backport, but if you do end up backporting patches after that, you
> should set the fix-version to be 2.6.1.
>
> Thanks
> +Vinod
>
>
> > On Nov 2, 2015, at 11:29 AM, Sangjin Lee <sj...@apache.org> wrote:
> >
> > As you may have seen, 2.6.2 is out
> > <http://markmail.org/thread/yw53xgz6wzpqnclt>. I have also retargeted
> all
> > open issues that were targeted for 2.6.2 to 2.6.3.
> >
> > Continuing the discussion in the email thread here
> > <http://markmail.org/thread/ofjlzurok223bzyi>, I'd like us to maintain
> the
> > cadence of monthly point releases in the 2.6.x line. It would be great if
> > we can have 2.6.3 released before the year-end holidays.
> >
> > If you have any bugfixes and improvements that are targeted for 2.7.x (or
> > 2.8) that you think are applicable to 2.6.x, please *set the target
> version
> > to 2.6.3* and merge them to branch-2.6. Please use your judgment in terms
> > of the applicability and quality of the changes so that we can ensure
> each
> > point release is consistently better quality than the previous one.
> Thanks
> > everyone!
> >
> > Regards,
> > Sangjin
>
>

Re: [DISCUSS] Looking to a 2.8.0 release

[Sangjin Lee]

I reviewed the current state of the YARN-2928 changes regarding its impact
if the timeline service v.2 is disabled. It does appear that there are a
lot of things that still do get created and enabled unconditionally
regardless of configuration. While this is understandable when we were
working to implement the feature, this clearly needs to be cleaned up so
that when disabled the timeline service v.2 doesn't impact other things.

I filed a JIRA for that work:
https://issues.apache.org/jira/browse/YARN-4356

We need to complete it before we can merge.

Somewhat related is the status of the configuration and what it means in
various contexts (client/app-side vs. server-side, v.1 vs. v.2, etc.). I
know there is an ongoing discussion regarding YARN-4183. We'll need to
reflect the outcome of that discussion.

My overall impression of whether this can be done for 2.8 is that it looks
rather challenging given the suggested timeframe. We also need to complete
several major tasks before it is ready.

Sangjin


On Wed, Nov 11, 2015 at 5:49 PM, Sangjin Lee <sjl...@gmail.com> wrote:

>
> On Wed, Nov 11, 2015 at 12:13 PM, Vinod Vavilapalli <
> vino...@hortonworks.com> wrote:
>
>> — YARN Timeline Service Next generation: YARN-2928: Lots of momentum,
>> but clearly a work in progress. Two options here
>> — If it is safe to ship it into 2.8 in a disable manner, we can
>> get the early code into trunk and all the way int o2.8.
>> — If it is not safe, it organically rolls over into 2.9
>>
>
> I'll review the changes on YARN-2928 to see what impact it has (if any) if
> the timeline service v.2 is disabled.
>
> Another condition for it to make 2.8 is whether the branch will be in a
> shape in a couple of weeks such that it adds value for folks that want to
> test it. Hopefully it will become clearer soon.
>
> Sangjin
>

Re: [DISCUSS] Looking to a 2.8.0 release

[Sangjin Lee]

On Wed, Nov 11, 2015 at 12:13 PM, Vinod Vavilapalli  wrote:

> — YARN Timeline Service Next generation: YARN-2928: Lots of momentum,
> but clearly a work in progress. Two options here
> — If it is safe to ship it into 2.8 in a disable manner, we can
> get the early code into trunk and all the way int o2.8.
> — If it is not safe, it organically rolls over into 2.9
>

I'll review the changes on YARN-2928 to see what impact it has (if any) if
the timeline service v.2 is disabled.

Another condition for it to make 2.8 is whether the branch will be in a
shape in a couple of weeks such that it adds value for folks that want to
test it. Hopefully it will become clearer soon.

Sangjin

[jira] [Created] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6546:
--

 Summary: reconcile the two versions of the timeline service 
performance tests
 Key: MAPREDUCE-6546
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: YARN-2928
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Priority: Minor


The trunk now has a version of the timeline service performance test 
(YARN-2556). The timeline service v.2 (YARN-2928) also has a performance test, 
and these two versions are quite similar (by design).

We need to reconcile the two.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6540) TestMRTimelineEventHandling fails

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6540:
--

 Summary: TestMRTimelineEventHandling fails
 Key: MAPREDUCE-6540
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6540
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Sangjin Lee
Assignee: Sangjin Lee


TestMRTimelineEventHandling fails after YARN-2859 is merged because it changed 
the port the AHS binds to in a mini cluster.

{noformat}
Running org.apache.hadoop.mapred.TestMRTimelineEventHandling
Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 184.38 sec <<< 
FAILURE! - in org.apache.hadoop.mapred.TestMRTimelineEventHandling
testMRTimelineEventHandling(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
  Time elapsed: 70.528 sec  <<< ERROR!
java.io.IOException: Job didn't finish in 30 seconds
at 
org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
at 
org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling(TestMRTimelineEventHandling.java:99)

testMapreduceJobTimelineServiceEnabled(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
  Time elapsed: 84.312 sec  <<< ERROR!
java.io.IOException: Job didn't finish in 30 seconds
at 
org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
at 
org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled(TestMRTimelineEventHandling.java:162)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

continuing releases on Apache Hadoop 2.6.x

[Sangjin Lee]

As you may have seen, 2.6.2 is out
. I have also retargeted all
open issues that were targeted for 2.6.2 to 2.6.3.

Continuing the discussion in the email thread here
, I'd like us to maintain the
cadence of monthly point releases in the 2.6.x line. It would be great if
we can have 2.6.3 released before the year-end holidays.

If you have any bugfixes and improvements that are targeted for 2.7.x (or
2.8) that you think are applicable to 2.6.x, please *set the target version
to 2.6.3* and merge them to branch-2.6. Please use your judgment in terms
of the applicability and quality of the changes so that we can ensure each
point release is consistently better quality than the previous one. Thanks
everyone!

Regards,
Sangjin

Re: [VOTE] Release Apache Hadoop 2.6.2

[Sangjin Lee]

Thanks Vinod! Thanks for the correction on the vote Andrew. Making some
rookie mistakes here. :)

On Mon, Oct 26, 2015 at 2:06 PM, Vinod Vavilapalli <vino...@hortonworks.com>
wrote:

> I was helping Sangjin offline with the release.
>
> We briefly discussed the KEYS problem before, but it missed my attention.
>
> I will get his KEYS committed right-away, the release is testable right
> away though.
>
> Regarding the voting period, let’s continue voting for two more days, the
> period also had the weekend during which a lot of people (atleast myself
> and team) didn’t pay attention to this vote.
>
> Thanks
> +Vinod
>
>
> > On Oct 26, 2015, at 1:50 PM, Andrew Wang <andrew.w...@cloudera.com>
> wrote:
> >
> > Hey Sangjin, did you add your release signing keys to the KEYS file? I
> > don't see it here:
> >
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Also only PMC votes are binding on releases, so I think we currently
> still
> > stand at 0 binding +1s.
> >
> > On Mon, Oct 26, 2015 at 1:28 PM, Sangjin Lee <sjl...@gmail.com> wrote:
> >
> >> That makes sense. Thanks for pointing that out. The git commit id is
> >> 0cfd050febe4a30b1ee1551dcc527589509fb681.
> >>
> >> On Mon, Oct 26, 2015 at 12:25 PM, Steve Loughran <
> ste...@hortonworks.com>
> >> wrote:
> >>
> >>>
> >>>> On 22 Oct 2015, at 22:14, Sangjin Lee <sj...@apache.org> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I have created a release candidate (RC0) for Hadoop 2.6.2.
> >>>>
> >>>> The RC is available at:
> >>> http://people.apache.org/~sjlee/hadoop-2.6.2-RC0/
> >>>>
> >>>> The RC tag in git is: release-2.6.2-RC0
> >>>>
> >>>
> >>> Tags can move; we should *never* vote for a release of one.
> >>>
> >>> What is the git commit #  ?
> >>>
> >>>> The list of JIRAs committed for 2.6.2:
> >>>>
> >>>
> >>
> https://issues.apache.org/jira/browse/YARN-4101?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%202.6.2
> >>>>
> >>>> The maven artifacts are staged at
> >>>>
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1022/
> >>>>
> >>>> Please try out the release candidate and vote. The vote will run for 5
> >>> days.
> >>>>
> >>>> Thanks,
> >>>> Sangjin
> >>>
> >>>
> >>
>
>

Re: [VOTE] Release Apache Hadoop 2.6.2

[Sangjin Lee]

Thanks Akira!

A friendly reminder to the community that the vote closes tomorrow. It
would be great if you could check out the release candidate and cast your
vote.

Sangjin

On Mon, Oct 26, 2015 at 5:00 AM, Akira AJISAKA <ajisa...@oss.nttdata.co.jp>
wrote:

> Thanks Sangjin for creating RC for 2.6.2. +1 (non-binding)
>
> - Verified signatures and checksums
> - Deployed a single node cluster
> - Built Tez 0.7.0 with Hadoop 2.6.2 pom
> - Built Hive 1.2.1 with Hadoop 2.6.2 pom
> - Ran some Hive on Tez queries successfully
>
> Regards,
> Akira
>
>
> On 10/23/15 06:14, Sangjin Lee wrote:
>
>> Hi all,
>>
>> I have created a release candidate (RC0) for Hadoop 2.6.2.
>>
>> The RC is available at: http://people.apache.org/~sjlee/hadoop-2.6.2-RC0/
>>
>> The RC tag in git is: release-2.6.2-RC0
>>
>> The list of JIRAs committed for 2.6.2:
>>
>> https://issues.apache.org/jira/browse/YARN-4101?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%202.6.2
>>
>> The maven artifacts are staged at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1022/
>>
>> Please try out the release candidate and vote. The vote will run for 5
>> days.
>>
>> Thanks,
>> Sangjin
>>
>>
>

[VOTE] Release Apache Hadoop 2.6.2

[Sangjin Lee]

Hi all,

I have created a release candidate (RC0) for Hadoop 2.6.2.

The RC is available at: http://people.apache.org/~sjlee/hadoop-2.6.2-RC0/

The RC tag in git is: release-2.6.2-RC0

The list of JIRAs committed for 2.6.2:
https://issues.apache.org/jira/browse/YARN-4101?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%202.6.2

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1022/

Please try out the release candidate and vote. The vote will run for 5 days.

Thanks,
Sangjin

Re: Planning for Apache Hadoop 2.6.2

[Sangjin Lee]

Another friendly reminder that I'll be cutting the branch and creating the
RC soon. I'm targeting tomorrow. Thanks!

Sangjin

On Mon, Oct 12, 2015 at 11:40 AM, Sangjin Lee <sj...@apache.org> wrote:

> Hi all,
>
> We are targeting next week to create the first RC for 2.6.2. Currently
> there are 14 issues that are committed to branch-2.6. If you have bug fixes
> that were made to branch-2 and/or branch-2.7 that are applicable to 2.6.x
> and would improve the quality of those releases, please take time to review
> them and commit them to branch-2.6.
>
> Also, if you have JIRAs that are targeted to 2.6.2 and are close to being
> done, this might be a good time to complete them.
>
> Do let me know if you have any questions.
>
> Thanks,
> Sangjin
>
> On Sat, Sep 26, 2015 at 9:19 AM, Sangjin Lee <sj...@apache.org> wrote:
>
>> I have updated branch-2.6 and force pushed the branch. I checked the
>> branch out from scratch and tested building it.
>>
>> Branch-2.6 is back open. If you have a patch ready for 2.6.2, double
>> check if it applies cleanly before committing to branch-2.6.
>>
>> Thanks,
>> Sangjin
>>
>> On Fri, Sep 25, 2015 at 4:42 PM, Sangjin Lee <sj...@apache.org> wrote:
>>
>>> Per Vinod's suggestion, in order to reduce the amount of movement I'll
>>> pick commits from branch-2.6 onto the tip of branch-2.6.1 rather than the
>>> other way around. This means I'll need to move branch-2.6 and force push
>>> that change.
>>>
>>> Could you please hold off on committing to branch-2.6 until I am done
>>> relocating the branch? I'll let you know when I'm done with that exercise.
>>> I expect I'll be done with this in 24 hours or so. Let me know if you have
>>> any concerns.
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Fri, Sep 25, 2015 at 12:53 PM, Sangjin Lee <sj...@apache.org> wrote:
>>>
>>>> Thanks folks. I'll get started on the items Vinod mentioned soon.
>>>>
>>>> If you have something you'd like to push for inclusion in 2.6.2, please
>>>> mark the target version as 2.6.2.
>>>>
>>>> I'd like to ask one more thing on top of that. It would be AWESOME if
>>>> you can check if it can be applied cleanly on top of 2.6.1, and if not,
>>>> provide an updated patch suitable for 2.6.1. This will help speed up the
>>>> 2.6.2 release process tremendously. If the person who's recommending the
>>>> JIRA for 2.6.2 inclusion can do it, that would be great. Help from the
>>>> original contributor might be helpful as well. Thanks for your cooperation!
>>>>
>>>> Regards,
>>>> Sangjin
>>>>
>>>> On Thu, Sep 24, 2015 at 9:39 PM, Akira AJISAKA <
>>>> ajisa...@oss.nttdata.co.jp> wrote:
>>>>
>>>>> Thanks Vinod and Sangjin for releasing 2.6.1 and starting discussion
>>>>> for 2.6.2!
>>>>>
>>>>> +1. If there's anything I can help you with, please tell me.
>>>>>
>>>>> Thanks,
>>>>> Akira
>>>>>
>>>>>
>>>>> On 9/25/15 13:23, Vinayakumar B wrote:
>>>>>
>>>>>> Thanks Vinod and Sangjin for making 2.6.1 release possible.
>>>>>>
>>>>>> Apologies for not getting time to verify and vote for the release.
>>>>>>
>>>>>> I will also be available to help for 2.6.2 if anything required.
>>>>>>
>>>>>> Thanks,
>>>>>> Vinay
>>>>>>
>>>>>> On Fri, Sep 25, 2015 at 12:16 AM, Vinod Vavilapalli <
>>>>>> vino...@hortonworks.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>
>>>>>> +1. Please take it over, I’ll standby for any help needed.
>>>>>>>
>>>>>>> Thanks
>>>>>>> +Vinod
>>>>>>>
>>>>>>>
>>>>>>> On Sep 24, 2015, at 11:34 AM, Sangjin Lee <sj...@apache.org>>>>>> sj...@apache.org>> wrote:
>>>>>>>
>>>>>>> I'd like to volunteer as the release manager for 2.6.2 unless there
>>>>>>> is an
>>>>>>> objection.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Planning for Apache Hadoop 2.6.2

[Sangjin Lee]

If you have backported bugfixes to 2.7.2, please take a moment to consider
if it is relevant (and important) for 2.6.x too, and if so backport it to
branch-2.6. Thanks!

On Tue, Oct 20, 2015 at 9:33 AM, Sangjin Lee <sj...@apache.org> wrote:

> Another friendly reminder that I'll be cutting the branch and creating the
> RC soon. I'm targeting tomorrow. Thanks!
>
> Sangjin
>
> On Mon, Oct 12, 2015 at 11:40 AM, Sangjin Lee <sj...@apache.org> wrote:
>
>> Hi all,
>>
>> We are targeting next week to create the first RC for 2.6.2. Currently
>> there are 14 issues that are committed to branch-2.6. If you have bug fixes
>> that were made to branch-2 and/or branch-2.7 that are applicable to 2.6.x
>> and would improve the quality of those releases, please take time to review
>> them and commit them to branch-2.6.
>>
>> Also, if you have JIRAs that are targeted to 2.6.2 and are close to being
>> done, this might be a good time to complete them.
>>
>> Do let me know if you have any questions.
>>
>> Thanks,
>> Sangjin
>>
>> On Sat, Sep 26, 2015 at 9:19 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>>> I have updated branch-2.6 and force pushed the branch. I checked the
>>> branch out from scratch and tested building it.
>>>
>>> Branch-2.6 is back open. If you have a patch ready for 2.6.2, double
>>> check if it applies cleanly before committing to branch-2.6.
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Fri, Sep 25, 2015 at 4:42 PM, Sangjin Lee <sj...@apache.org> wrote:
>>>
>>>> Per Vinod's suggestion, in order to reduce the amount of movement I'll
>>>> pick commits from branch-2.6 onto the tip of branch-2.6.1 rather than the
>>>> other way around. This means I'll need to move branch-2.6 and force push
>>>> that change.
>>>>
>>>> Could you please hold off on committing to branch-2.6 until I am done
>>>> relocating the branch? I'll let you know when I'm done with that exercise.
>>>> I expect I'll be done with this in 24 hours or so. Let me know if you have
>>>> any concerns.
>>>>
>>>> Thanks,
>>>> Sangjin
>>>>
>>>> On Fri, Sep 25, 2015 at 12:53 PM, Sangjin Lee <sj...@apache.org> wrote:
>>>>
>>>>> Thanks folks. I'll get started on the items Vinod mentioned soon.
>>>>>
>>>>> If you have something you'd like to push for inclusion in 2.6.2,
>>>>> please mark the target version as 2.6.2.
>>>>>
>>>>> I'd like to ask one more thing on top of that. It would be AWESOME if
>>>>> you can check if it can be applied cleanly on top of 2.6.1, and if not,
>>>>> provide an updated patch suitable for 2.6.1. This will help speed up the
>>>>> 2.6.2 release process tremendously. If the person who's recommending the
>>>>> JIRA for 2.6.2 inclusion can do it, that would be great. Help from the
>>>>> original contributor might be helpful as well. Thanks for your 
>>>>> cooperation!
>>>>>
>>>>> Regards,
>>>>> Sangjin
>>>>>
>>>>> On Thu, Sep 24, 2015 at 9:39 PM, Akira AJISAKA <
>>>>> ajisa...@oss.nttdata.co.jp> wrote:
>>>>>
>>>>>> Thanks Vinod and Sangjin for releasing 2.6.1 and starting discussion
>>>>>> for 2.6.2!
>>>>>>
>>>>>> +1. If there's anything I can help you with, please tell me.
>>>>>>
>>>>>> Thanks,
>>>>>> Akira
>>>>>>
>>>>>>
>>>>>> On 9/25/15 13:23, Vinayakumar B wrote:
>>>>>>
>>>>>>> Thanks Vinod and Sangjin for making 2.6.1 release possible.
>>>>>>>
>>>>>>> Apologies for not getting time to verify and vote for the release.
>>>>>>>
>>>>>>> I will also be available to help for 2.6.2 if anything required.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Vinay
>>>>>>>
>>>>>>> On Fri, Sep 25, 2015 at 12:16 AM, Vinod Vavilapalli <
>>>>>>> vino...@hortonworks.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>> +1. Please take it over, I’ll standby for any help needed.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> +Vinod
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sep 24, 2015, at 11:34 AM, Sangjin Lee <sj...@apache.org>>>>>>> sj...@apache.org>> wrote:
>>>>>>>>
>>>>>>>> I'd like to volunteer as the release manager for 2.6.2 unless there
>>>>>>>> is an
>>>>>>>> objection.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: [DISCUSS] About the details of JDK-8 support

[Sangjin Lee]

+1

On Thu, Oct 15, 2015 at 7:57 AM, Steve Loughran 
wrote:

>
> > On 15 Oct 2015, at 14:42, Karthik Kambatla  wrote:
> >
> > On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer 
> wrote:
> >
> >>
> >> If people want, I could setup a cut off of yetus master to run the
> jenkins
> >> test-patch.  (multiple maven repos, docker support, multijdk support, …
> )
> >> Yetus would get some real world testing out of it and hadoop common-dev
> >> could stop spinning in circles over some of the same issues month after
> >> month. ;)
> >>
> >
> > Seems like a step in the right direction.
> >
> > Should we expect a downtime and is there a good/bad time to do this?
>
>
>
> Weekends are the times to work on Jenkins
>
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before
> monday morning PST.
>
> Two days time, then?

Re: Planning for Apache Hadoop 2.6.2

[Sangjin Lee]

Hi all,

We are targeting next week to create the first RC for 2.6.2. Currently
there are 14 issues that are committed to branch-2.6. If you have bug fixes
that were made to branch-2 and/or branch-2.7 that are applicable to 2.6.x
and would improve the quality of those releases, please take time to review
them and commit them to branch-2.6.

Also, if you have JIRAs that are targeted to 2.6.2 and are close to being
done, this might be a good time to complete them.

Do let me know if you have any questions.

Thanks,
Sangjin

On Sat, Sep 26, 2015 at 9:19 AM, Sangjin Lee <sj...@apache.org> wrote:

> I have updated branch-2.6 and force pushed the branch. I checked the
> branch out from scratch and tested building it.
>
> Branch-2.6 is back open. If you have a patch ready for 2.6.2, double check
> if it applies cleanly before committing to branch-2.6.
>
> Thanks,
> Sangjin
>
> On Fri, Sep 25, 2015 at 4:42 PM, Sangjin Lee <sj...@apache.org> wrote:
>
>> Per Vinod's suggestion, in order to reduce the amount of movement I'll
>> pick commits from branch-2.6 onto the tip of branch-2.6.1 rather than the
>> other way around. This means I'll need to move branch-2.6 and force push
>> that change.
>>
>> Could you please hold off on committing to branch-2.6 until I am done
>> relocating the branch? I'll let you know when I'm done with that exercise.
>> I expect I'll be done with this in 24 hours or so. Let me know if you have
>> any concerns.
>>
>> Thanks,
>> Sangjin
>>
>> On Fri, Sep 25, 2015 at 12:53 PM, Sangjin Lee <sj...@apache.org> wrote:
>>
>>> Thanks folks. I'll get started on the items Vinod mentioned soon.
>>>
>>> If you have something you'd like to push for inclusion in 2.6.2, please
>>> mark the target version as 2.6.2.
>>>
>>> I'd like to ask one more thing on top of that. It would be AWESOME if
>>> you can check if it can be applied cleanly on top of 2.6.1, and if not,
>>> provide an updated patch suitable for 2.6.1. This will help speed up the
>>> 2.6.2 release process tremendously. If the person who's recommending the
>>> JIRA for 2.6.2 inclusion can do it, that would be great. Help from the
>>> original contributor might be helpful as well. Thanks for your cooperation!
>>>
>>> Regards,
>>> Sangjin
>>>
>>> On Thu, Sep 24, 2015 at 9:39 PM, Akira AJISAKA <
>>> ajisa...@oss.nttdata.co.jp> wrote:
>>>
>>>> Thanks Vinod and Sangjin for releasing 2.6.1 and starting discussion
>>>> for 2.6.2!
>>>>
>>>> +1. If there's anything I can help you with, please tell me.
>>>>
>>>> Thanks,
>>>> Akira
>>>>
>>>>
>>>> On 9/25/15 13:23, Vinayakumar B wrote:
>>>>
>>>>> Thanks Vinod and Sangjin for making 2.6.1 release possible.
>>>>>
>>>>> Apologies for not getting time to verify and vote for the release.
>>>>>
>>>>> I will also be available to help for 2.6.2 if anything required.
>>>>>
>>>>> Thanks,
>>>>> Vinay
>>>>>
>>>>> On Fri, Sep 25, 2015 at 12:16 AM, Vinod Vavilapalli <
>>>>> vino...@hortonworks.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>
>>>>> +1. Please take it over, I’ll standby for any help needed.
>>>>>>
>>>>>> Thanks
>>>>>> +Vinod
>>>>>>
>>>>>>
>>>>>> On Sep 24, 2015, at 11:34 AM, Sangjin Lee <sj...@apache.org>>>>> sj...@apache.org>> wrote:
>>>>>>
>>>>>> I'd like to volunteer as the release manager for 2.6.2 unless there
>>>>>> is an
>>>>>> objection.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: [DISCUSS] About the details of JDK-8 support

[Sangjin Lee]

Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
source-wise) **at this point** would be an issue. I concur with the
sentiment that we should preserve the java 7 support on branch-2 (not not
move to java 8 source level) but can consider it for trunk. My 2 cents.

Thanks,
Sangjin

On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran 
wrote:

>
> > On 7 Oct 2015, at 22:39, Elliott Clark  wrote:
> >
> > On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa  wrote:
> >
> >> Do you have any concern about this? I’ve not
> >> tested with HBase yet.
> >>
> >
> > We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
> > quite a while. Everything has been very stable for us. We're running and
> > compiling with jdk8.
> >
> > We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
> > didn't do too well.
> >
>
> maybe related to the initial memory requirements being higher?
>
> otherwise: did you file a JIRA?
>
>
> > I'd be +1 on dropping jdk7 support. However as downstream developer it
> > would be very weird for that to happen on anything but a major release.
>
>
> Past discussion (including a big contrib from twitter) was that breaking
> Java 7 support breaks all client apps too, which is not something people
> were ready for.
>
> I'd  be +1 for moving trunk to the java-8 compatible dependencies, and to
> have the jenkins nighly builds only before java 8; we'd still have the
> patch and branch-2 runs on Java 7. That way, people will only have one
> nightly mail from Jenkins saying the build is broken, and maybe pay more
> attention to the fact.

Re: Planning for Apache Hadoop 2.6.2

[Sangjin Lee]

I have updated branch-2.6 and force pushed the branch. I checked the branch
out from scratch and tested building it.

Branch-2.6 is back open. If you have a patch ready for 2.6.2, double check
if it applies cleanly before committing to branch-2.6.

Thanks,
Sangjin

On Fri, Sep 25, 2015 at 4:42 PM, Sangjin Lee <sj...@apache.org> wrote:

> Per Vinod's suggestion, in order to reduce the amount of movement I'll
> pick commits from branch-2.6 onto the tip of branch-2.6.1 rather than the
> other way around. This means I'll need to move branch-2.6 and force push
> that change.
>
> Could you please hold off on committing to branch-2.6 until I am done
> relocating the branch? I'll let you know when I'm done with that exercise.
> I expect I'll be done with this in 24 hours or so. Let me know if you have
> any concerns.
>
> Thanks,
> Sangjin
>
> On Fri, Sep 25, 2015 at 12:53 PM, Sangjin Lee <sj...@apache.org> wrote:
>
>> Thanks folks. I'll get started on the items Vinod mentioned soon.
>>
>> If you have something you'd like to push for inclusion in 2.6.2, please
>> mark the target version as 2.6.2.
>>
>> I'd like to ask one more thing on top of that. It would be AWESOME if you
>> can check if it can be applied cleanly on top of 2.6.1, and if not, provide
>> an updated patch suitable for 2.6.1. This will help speed up the 2.6.2
>> release process tremendously. If the person who's recommending the JIRA for
>> 2.6.2 inclusion can do it, that would be great. Help from the original
>> contributor might be helpful as well. Thanks for your cooperation!
>>
>> Regards,
>> Sangjin
>>
>> On Thu, Sep 24, 2015 at 9:39 PM, Akira AJISAKA <
>> ajisa...@oss.nttdata.co.jp> wrote:
>>
>>> Thanks Vinod and Sangjin for releasing 2.6.1 and starting discussion for
>>> 2.6.2!
>>>
>>> +1. If there's anything I can help you with, please tell me.
>>>
>>> Thanks,
>>> Akira
>>>
>>>
>>> On 9/25/15 13:23, Vinayakumar B wrote:
>>>
>>>> Thanks Vinod and Sangjin for making 2.6.1 release possible.
>>>>
>>>> Apologies for not getting time to verify and vote for the release.
>>>>
>>>> I will also be available to help for 2.6.2 if anything required.
>>>>
>>>> Thanks,
>>>> Vinay
>>>>
>>>> On Fri, Sep 25, 2015 at 12:16 AM, Vinod Vavilapalli <
>>>> vino...@hortonworks.com
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> +1. Please take it over, I’ll standby for any help needed.
>>>>>
>>>>> Thanks
>>>>> +Vinod
>>>>>
>>>>>
>>>>> On Sep 24, 2015, at 11:34 AM, Sangjin Lee <sj...@apache.org>>>> sj...@apache.org>> wrote:
>>>>>
>>>>> I'd like to volunteer as the release manager for 2.6.2 unless there is
>>>>> an
>>>>> objection.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Planning for Apache Hadoop 2.6.2

[Sangjin Lee]

Thanks folks. I'll get started on the items Vinod mentioned soon.

If you have something you'd like to push for inclusion in 2.6.2, please
mark the target version as 2.6.2.

I'd like to ask one more thing on top of that. It would be AWESOME if you
can check if it can be applied cleanly on top of 2.6.1, and if not, provide
an updated patch suitable for 2.6.1. This will help speed up the 2.6.2
release process tremendously. If the person who's recommending the JIRA for
2.6.2 inclusion can do it, that would be great. Help from the original
contributor might be helpful as well. Thanks for your cooperation!

Regards,
Sangjin

On Thu, Sep 24, 2015 at 9:39 PM, Akira AJISAKA <ajisa...@oss.nttdata.co.jp>
wrote:

> Thanks Vinod and Sangjin for releasing 2.6.1 and starting discussion for
> 2.6.2!
>
> +1. If there's anything I can help you with, please tell me.
>
> Thanks,
> Akira
>
>
> On 9/25/15 13:23, Vinayakumar B wrote:
>
>> Thanks Vinod and Sangjin for making 2.6.1 release possible.
>>
>> Apologies for not getting time to verify and vote for the release.
>>
>> I will also be available to help for 2.6.2 if anything required.
>>
>> Thanks,
>> Vinay
>>
>> On Fri, Sep 25, 2015 at 12:16 AM, Vinod Vavilapalli <
>> vino...@hortonworks.com
>>
>>> wrote:
>>>
>>
>> +1. Please take it over, I’ll standby for any help needed.
>>>
>>> Thanks
>>> +Vinod
>>>
>>>
>>> On Sep 24, 2015, at 11:34 AM, Sangjin Lee <sj...@apache.org>> sj...@apache.org>> wrote:
>>>
>>> I'd like to volunteer as the release manager for 2.6.2 unless there is an
>>> objection.
>>>
>>>
>>>
>>
>

Re: Planning for Apache Hadoop 2.6.2

[Sangjin Lee]

Per Vinod's suggestion, in order to reduce the amount of movement I'll pick
commits from branch-2.6 onto the tip of branch-2.6.1 rather than the other
way around. This means I'll need to move branch-2.6 and force push that
change.

Could you please hold off on committing to branch-2.6 until I am done
relocating the branch? I'll let you know when I'm done with that exercise.
I expect I'll be done with this in 24 hours or so. Let me know if you have
any concerns.

Thanks,
Sangjin

On Fri, Sep 25, 2015 at 12:53 PM, Sangjin Lee <sj...@apache.org> wrote:

> Thanks folks. I'll get started on the items Vinod mentioned soon.
>
> If you have something you'd like to push for inclusion in 2.6.2, please
> mark the target version as 2.6.2.
>
> I'd like to ask one more thing on top of that. It would be AWESOME if you
> can check if it can be applied cleanly on top of 2.6.1, and if not, provide
> an updated patch suitable for 2.6.1. This will help speed up the 2.6.2
> release process tremendously. If the person who's recommending the JIRA for
> 2.6.2 inclusion can do it, that would be great. Help from the original
> contributor might be helpful as well. Thanks for your cooperation!
>
> Regards,
> Sangjin
>
> On Thu, Sep 24, 2015 at 9:39 PM, Akira AJISAKA <ajisa...@oss.nttdata.co.jp
> > wrote:
>
>> Thanks Vinod and Sangjin for releasing 2.6.1 and starting discussion for
>> 2.6.2!
>>
>> +1. If there's anything I can help you with, please tell me.
>>
>> Thanks,
>> Akira
>>
>>
>> On 9/25/15 13:23, Vinayakumar B wrote:
>>
>>> Thanks Vinod and Sangjin for making 2.6.1 release possible.
>>>
>>> Apologies for not getting time to verify and vote for the release.
>>>
>>> I will also be available to help for 2.6.2 if anything required.
>>>
>>> Thanks,
>>> Vinay
>>>
>>> On Fri, Sep 25, 2015 at 12:16 AM, Vinod Vavilapalli <
>>> vino...@hortonworks.com
>>>
>>>> wrote:
>>>>
>>>
>>> +1. Please take it over, I’ll standby for any help needed.
>>>>
>>>> Thanks
>>>> +Vinod
>>>>
>>>>
>>>> On Sep 24, 2015, at 11:34 AM, Sangjin Lee <sj...@apache.org>>> sj...@apache.org>> wrote:
>>>>
>>>> I'd like to volunteer as the release manager for 2.6.2 unless there is
>>>> an
>>>> objection.
>>>>
>>>>
>>>>
>>>
>>
>

Re: Planning for Apache Hadoop 2.6.2

[Sangjin Lee]

Thanks Vinod for starting the discussion for 2.6.2!

I'd like to volunteer as the release manager for 2.6.2 unless there is an
objection. As I've worked with Vinod and Akira for 2.6.1 and triaged a
number of issues, I feel I should be in a good position to take care of
2.6.2. Let me know your thoughts.

Regards,
Sangjin

On Thu, Sep 24, 2015 at 10:47 AM, Vinod Kumar Vavilapalli <
vino...@apache.org> wrote:

> Hi all,
>
>
> Now that 2.6.1 is done, it’s time to look forward to 2.6.2. Things to take
> care of:
>
>
>  (#1) Branch:
>
> — The first step in this is getting the branch ready.
>
> — As I mentioned before [1], the plan is to rebase branch-2.6 based off
> branch-2.6.1 i.e. start with 2.6.1 + add patches that are already in
> today’s branch-2.6. This is because, as I noted before, the other way
> around of applying 150 odd 2.6.1 patches on today’s 2.6 is very cumbersome.
>
> — Unless I hear otherwise, we will start this process next week.
>
>
>  (#2) Release:
>
> — My current idea is to consistently roll out a point release every
> month, so for 2.6.2, we can target an RC exactly a month from now (2.6.1
> release date)
>
>
>  (#3) Patches
>
> — 2.6.2 will have the few patches that committers already put into
> branch-2.6. In addition, we should triage and analyze tickets that are
> already marked [2] against 2.6.2.
>
> — Thoughts / inputs welcome on any patches that you want in 2.6.2.
> Please use “Target Version = 2.6.2” if you want any patch in the release.
>
>
> Thanks,
>
> +Vinod
>
>
> [1] Re: [VOTE] Release Apache Hadoop 2.6.1 RC0
> http://markmail.org/message/co6m5n6zmdpzsdok
>
> [2] 2.6.2 Target tickets:
> https://issues.apache.org/jira/issues/?filter=12333460
>

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

[Sangjin Lee]

+1 (non-binding)

Verified the signatures, set up a pseudo-distributed cluster, ran several
test jobs, and ran an uber job. Also verified that the UI issue I saw on
RC0 is now gone. Thanks Vinod!

Sangjin

On Thu, Sep 17, 2015 at 7:24 PM, Jian He <j...@hortonworks.com> wrote:

> +1 (binding)
>
> Build from source code.
> Deployed a local cluster.
> Validated sample jobs passed.
>
> Jian
>
> > On Sep 18, 2015, at 7:34 AM, Wangda Tan <wheele...@gmail.com> wrote:
> >
> > Deployed a local cluster, verified configured cluster with node labels,
> run
> > jobs with/without node labels.
> >
> > +1 (non-binding)
> >
> > Thanks!
> >
> > On Thu, Sep 17, 2015 at 2:40 PM, Xuan Gong <xg...@hortonworks.com>
> wrote:
> >
> >> Update my vote from +1 (non-binding) to +1 binding
> >>
> >> Thanks
> >>
> >> Xuan Gong
> >>
> >>> On Sep 17, 2015, at 2:05 PM, Xuan Gong <xg...@hortonworks.com> wrote:
> >>>
> >>> +1 (non-binding)
> >>> Download and compile the source code, run several MR jobs.
> >>>
> >>> Xuan Gong
> >>>
> >>>> On Sep 16, 2015, at 7:10 PM, Vinod Kumar Vavilapalli <
> >> vino...@apache.org> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> After a nearly month long [1] toil, with loads of help from Sangjin
> Lee
> >> and
> >>>> Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a
> >> release
> >>>> candidate RC1 for hadoop-2.6.1.
> >>>>
> >>>> RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI
> >> fixes
> >>>> for the issue Sangjin raised (YARN-3171 and the dependencies
> YARN-3779,
> >>>> YARN-3248), additional fix to avoid incompatibility (YARN-3740), other
> >> UI
> >>>> bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch
> >> for
> >>>> YARN-2890) that Jeff Zhang raised.
> >>>>
> >>>> The RC is available at:
> >> http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/
> >>>>
> >>>> The RC tag in git is: release-2.6.1-RC1
> >>>>
> >>>> The maven artifacts are available via repository.apache.org at
> >>>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1021
> >>>>
> >>>> Some notes from our release process
> >>>> -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
> >>>> non-committed but desired patches. 2.6.1 is already big as is and is
> >> late
> >>>> by any standard, we can definitely include them in the next release.
> >>>> - The 2.6.1 wiki page [3] captures some (but not all) of the context
> of
> >>>> the patches that we pushed in.
> >>>> - Given the number of fixes pushed [4] in, we had to make a bunch of
> >>>> changes to our original plan - we added a few improvements that helped
> >> us
> >>>> backport patches easier (or in many cases made backports possible),
> and
> >> we
> >>>> dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
> >>>> HDFS-7611, HDFS-7843, HDFS-8850).
> >>>> - I ran all the unit tests which (surprisingly?) passed. (Except for
> >> one,
> >>>> which pointed out a missing fix HDFS-7552).
> >>>>
> >>>> As discussed before [5]
> >>>> - This release is the first point release after 2.6.0
> >>>> - I’d like to use this as a starting release for 2.6.2 in a few weeks
> >> and
> >>>> then follow up with more of these.
> >>>>
> >>>> Please try the release and vote; the vote will run for the usual 5
> days.
> >>>>
> >>>> Thanks,
> >>>> Vinod
> >>>>
> >>>> [0] Hadoop 2.6.1 RC0 vote:
> http://markmail.org/thread/ubut2rn3lodc55iy
> >>>> [1] Hadoop 2.6.1 Release process thread:
> >>>> http://markmail.org/thread/wkbgkxkhntx5tlux
> >>>> [2] 2.6.1 Pending tickets:
> >>>> https://issues.apache.org/jira/issues/?filter=12331711
> >>>> [3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1
> >>>> -Working-Notes
> >>>> [4] List of 2.6.1 patches pushed:
> >>>> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1
> >>>> %20and%20labels%20%3D%20%222.6.1-candidate%22
> >>>> [5] Planning Hadoop 2.6.1 release:
> >>>> http://markmail.org/thread/sbykjn5xgnksh6wg
> >>>>
> >>>> PS:
> >>>> - Note that branch-2.6 which will be the base for 2.6.2 doesn't have
> >> these
> >>>> fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based
> >> off
> >>>> 2.6.1.
> >>>> - The additional patches in RC1 that got into 2.6.1 all the way from
> 2.8
> >>>> are NOT in 2.7.2 yet, this will be done as a followup.
> >>>
> >>
> >>
>
>

Re: [VOTE] Release Apache Hadoop 2.6.1 RC0

[Sangjin Lee]

I verified the signatures for both source and the binary tarballs. I
started up a pseudo-distributed cluster, and tested simple apps such as
sleep and terasort.

I do see one issue with the RM UI where the sorting by id is broken. The
table is not rendered in the expected id-descending order, and when I click
the sort control, nothing happens. Sorting by other columns works fine.

Is anyone else able to reproduce the issue? I checked 2.6.0, and it works
fine on 2.6.0.

On Wed, Sep 9, 2015 at 6:00 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
wrote:

> Hi all,
>
> After a nearly month long [1] toil, with loads of help from Sangjin Lee
> and Akira Ajisaka, and 153 commits later, I've created a release candidate
> RC0 for hadoop-2.6.1.
>
> The RC is available at:
> http://people.apache.org/~vinodkv/hadoop-2.6.1-RC0/
>
> The RC tag in git is: release-2.6.1-RC0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1020
>
> Some notes from our release process
>  -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
> non-committed but desired patches. 2.6.1 is already big as is and is late
> by any standard, we can definitely include them in the next release.
>  - The 2.6.1 wiki page [3] captures some (but not all) of the context of
> the patches that we pushed in.
>  - Given the number of fixes pushed [4] in, we had to make a bunch of
> changes to our original plan - we added a few improvements that helped us
> backport patches easier (or in many cases made backports possible), and we
> dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
> HDFS-7611, HDFS-7843, HDFS-8850).
>  - I ran all the unit tests which (surprisingly?) passed. (Except for one,
> which pointed out a missing fix HDFS-7552).
>
> As discussed before [5]
>  - This release is the first point release after 2.6.0
>  - I’d like to use this as a starting release for 2.6.2 in a few weeks
> and then follow up with more of these.
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] Hadoop 2.6.1 Release process thread:
> http://markmail.org/thread/wkbgkxkhntx5tlux
> [2] 2.6.1 Pending tickets:
> https://issues.apache.org/jira/issues/?filter=12331711
> [3] 2.6.1 Wiki page:
> https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes
> [4] List of 2.6.1 patches pushed:
> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1%20and%20labels%20%3D%20%222.6.1-candidate%22
> [5] Planning Hadoop 2.6.1 release:
> http://markmail.org/thread/sbykjn5xgnksh6wg
>
> PS:
>  - Note that branch-2.6 which will be the base for 2.6.2 doesn't have
> these fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based
> off 2.6.1.
>  - Patches that got into 2.6.1 all the way from 2.8 are NOT in 2.7.2 yet,
> this will be done as a followup.
>
>

Re: Hadoop 2.6.1 Release process thread

[Sangjin Lee]

It might have been because we thought that HDFS-7704 was going to make it.
It's both make it or neither does. Now that we know HDFS-7704 is out,
HDFS-7916 should definitely be out. I hope that clarifies.

On Tue, Aug 11, 2015 at 6:26 PM, Vinod Kumar Vavilapalli 
vino...@hortonworks.com wrote:

 I earlier removed HDFS-7916 from the list given HDFS-7704 was only in
 2.7.0.

 Chris Trezzo added it back and so it appeared in my lists.

 I removed it again, Chris, please comment on why you added it back. If you
 want it included, please comment here and we can add it after we figure out
 the why and the dependent tickets.

 Thanks
 +Vinod

 On Aug 11, 2015, at 4:37 PM, Sangjin Lee sjl...@gmail.commailto:
 sjl...@gmail.com wrote:

 Could you double check HDFS-7916? HDFS-7916 is needed only if HDFS-7704
 makes it. However, I see commits for HDFS-7916 in this list, but not for
 HDFS-7704. If HDFS-7704 is not in the list, we should not backport
 HDFS-7916 as it fixes an issue introduced by HDFS-7704.

 On Tue, Aug 11, 2015 at 4:10 PM, Vinod Kumar Vavilapalli 
 vino...@hortonworks.commailto:vino...@hortonworks.com wrote:
 Put the list here:
 https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes. And started
 figuring out ways to fast-path the cherry-picks.

 Thanks
 +Vinod

 On Aug 11, 2015, at 1:15 PM, Vinod Kumar Vavilapalli vino...@apache.org
 mailto:vino...@apache.org wrote:

   (2) With Wangda's help offline, I prepared an ordered list of
 cherry-pick commits that we can do from our candidate list [1], will do
 some ground work today.

Re: Planning Hadoop 2.6.1 release

[Sangjin Lee]

See my later update in the thread. HDFS-7704 is in the list.

Thanks,
Sangjin

On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli 
vino...@hortonworks.com wrote:

 Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and is
 not part of the candidate list. Removed HDFS-7916 from the list.

 Thanks
 +Vinod

  On Jul 24, 2015, at 6:32 PM, Sangjin Lee sj...@apache.org wrote:
 
  Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
  applies to 2.6.
 
  Thanks,
  Sangjin

Re: Planning Hadoop 2.6.1 release

[Sangjin Lee]

Thanks Akira.

I'd like to make one small correction. If we're getting HDFS-7704, then we
should also get HDFS-7916. My earlier comment was assuming HDFS-7704 was
not included in the list. But if is (and I think it should), then we should
also get HDFS-7916 as it addresses an important issue related to HDFS-7704.
Hope it makes it clear.

Sangjin

On Fri, Jul 31, 2015 at 10:01 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp
wrote:

 Thanks Joep and your team members for creating the list. I really
 appreciate your work. I looked your 'not yet marked with 2.6.1-candidate'
 list and categorized them.

 1) Now marked as 2.6.1-candidate and I agree with you to keep it marked.

 * HDFS-7213
 * HDFS-7788
 * HDFS-7884
 * HDFS-7930
 * YARN-2856
 * YARN-3222
 * YARN-3238
 * YARN-3464
 * YARN-3526
 * YARN-3850

 Thanks Sangjin for creating patches for branch-2.6.

 2) Not yet marked as 2.6.1-candidate that I'd like to see in 2.6.1

 * HDFS-7182
 * HDFS-7314
 * HDFS-7704
 * HDFS-7894
 * HDFS-7929 and HDFS-8480 (they are related)
 * HDFS-7980
 * HDFS-8245
 * HDFS-8270
 * HDFS-8404
 * HDFS-8486
 * MAPREDUCE-5465
 * MAPREDUCE-5649
 * MAPREDUCE-6166
 * MAPREDUCE-6238
 * MAPREDUCE-6300
 * YARN-2952
 * YARN-2997
 * YARN-3094
 * YARN-3176 (to be fixed soon, I think)
 * YARN-3231
 * HADOOP-11295
 * HADOOP-11812

 3) Not yet marked as 2.6.1-candidate. I'd like to drop

 * HDFS-7281 (incompatible change)
 * HDFS-7446 (this looks to be an improvement)
 * HDFS-7916 (cannot apply to branch-2.6 as Sangjin mentioned)

 Hi Vinod, could you mark the issues in 2) as 2.6.1-candidate?

 I'd like to freeze the candidate list in about 7 days and start
 backporting them. Do you have any thoughts?

 Regards,
 Akira


 On 7/25/15 10:32, Sangjin Lee wrote:

 Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
 applies to 2.6.

 Thanks,
 Sangjin

 On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli 
 vino...@hortonworks.com wrote:

 I’ve added them all to the 2.6.1-candidate list. I included everything
 even though some of them are major tickets. The list is getting large, we
 will have to cut these down once we get down to the next phase of
 figuring
 out what to include and what not to.

 Thanks
 +Vinod

 On Jul 21, 2015, at 2:15 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp

 wrote:


 Thanks Vinod for updating the candidate list.
 I'd like to include the followings 12 JIRAs:

 * YARN-3641
 * YARN-3585
 * YARN-2910
 * HDFS-8431
 * HDFS-7830
 * HDFS-7763
 * HDFS-7742
 * HDFS-7235
 * HDFS-7225
 * MAPREDUCE-6324
 * HADOOP-11934
 * HADOOP-11491

 Thanks,
 Akira

 On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:

   - I also have a bunch of patches that I’d like to include, will
 update

 them right away.


 I’ve just finished this. The latest 2.6.1-candidate list is up at 64

 JIRAs.


 Others, please look at the list and post anything else you’d like to

 get included for 2.6.1.


 Thanks
 +Vinod


 On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli 

 vino...@hortonworks.commailto:vino...@hortonworks.com wrote:


 Alright, I’d like to make progress while the issue is hot.

 I created a label to discuss on the candidate list of patches:


 https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate
 
 https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate


 Next steps, I’ll do the following
   - Review 2.7 and 2.8 blocker/critical tickets and see what makes
 sense

 for 2.6.1 and add as candidates

   - I haven’t reviewed the current list yet, the seed list is from this

 email thread. Will review them.

   - I also have a bunch of patches that I’d like to include, will update

 them right away.


 Others, please look at the current list and let me know what else you’d

 like to include.


 I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week

 and then start the release process. @Akira, let’s sync up offline on
 how to
 take this forward in terms of the release process.


 Thanks
 +Vinod

Re: Planning Hadoop 2.6.1 release

[Sangjin Lee]

Just for the completeness, the following JIRAs have already been committed
to branch-2.6 and thus will be part of 2.6.1:

HADOOP-11307
YARN-2375
HDFS-7425
HDFS-4882
HDFS-7489
HDFS-7503
HDFS-7443
HDFS-3443
HADOOP-11466
HDFS-7733
MAPREDUCE-6237
YARN-3251
HDFS-6153 reverted


On Fri, Jul 31, 2015 at 10:07 AM, Sangjin Lee sj...@apache.org wrote:

 Thanks Akira.

 I'd like to make one small correction. If we're getting HDFS-7704, then we
 should also get HDFS-7916. My earlier comment was assuming HDFS-7704 was
 not included in the list. But if is (and I think it should), then we should
 also get HDFS-7916 as it addresses an important issue related to HDFS-7704.
 Hope it makes it clear.

 Sangjin

 On Fri, Jul 31, 2015 at 10:01 AM, Akira AJISAKA 
 ajisa...@oss.nttdata.co.jp wrote:

 Thanks Joep and your team members for creating the list. I really
 appreciate your work. I looked your 'not yet marked with 2.6.1-candidate'
 list and categorized them.

 1) Now marked as 2.6.1-candidate and I agree with you to keep it marked.

 * HDFS-7213
 * HDFS-7788
 * HDFS-7884
 * HDFS-7930
 * YARN-2856
 * YARN-3222
 * YARN-3238
 * YARN-3464
 * YARN-3526
 * YARN-3850

 Thanks Sangjin for creating patches for branch-2.6.

 2) Not yet marked as 2.6.1-candidate that I'd like to see in 2.6.1

 * HDFS-7182
 * HDFS-7314
 * HDFS-7704
 * HDFS-7894
 * HDFS-7929 and HDFS-8480 (they are related)
 * HDFS-7980
 * HDFS-8245
 * HDFS-8270
 * HDFS-8404
 * HDFS-8486
 * MAPREDUCE-5465
 * MAPREDUCE-5649
 * MAPREDUCE-6166
 * MAPREDUCE-6238
 * MAPREDUCE-6300
 * YARN-2952
 * YARN-2997
 * YARN-3094
 * YARN-3176 (to be fixed soon, I think)
 * YARN-3231
 * HADOOP-11295
 * HADOOP-11812

 3) Not yet marked as 2.6.1-candidate. I'd like to drop

 * HDFS-7281 (incompatible change)
 * HDFS-7446 (this looks to be an improvement)
 * HDFS-7916 (cannot apply to branch-2.6 as Sangjin mentioned)

 Hi Vinod, could you mark the issues in 2) as 2.6.1-candidate?

 I'd like to freeze the candidate list in about 7 days and start
 backporting them. Do you have any thoughts?

 Regards,
 Akira


 On 7/25/15 10:32, Sangjin Lee wrote:

 Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
 applies to 2.6.

 Thanks,
 Sangjin

 On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli 
 vino...@hortonworks.com wrote:

 I’ve added them all to the 2.6.1-candidate list. I included everything
 even though some of them are major tickets. The list is getting large,
 we
 will have to cut these down once we get down to the next phase of
 figuring
 out what to include and what not to.

 Thanks
 +Vinod

 On Jul 21, 2015, at 2:15 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp

 wrote:


 Thanks Vinod for updating the candidate list.
 I'd like to include the followings 12 JIRAs:

 * YARN-3641
 * YARN-3585
 * YARN-2910
 * HDFS-8431
 * HDFS-7830
 * HDFS-7763
 * HDFS-7742
 * HDFS-7235
 * HDFS-7225
 * MAPREDUCE-6324
 * HADOOP-11934
 * HADOOP-11491

 Thanks,
 Akira

 On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:

   - I also have a bunch of patches that I’d like to include, will
 update

 them right away.


 I’ve just finished this. The latest 2.6.1-candidate list is up at 64

 JIRAs.


 Others, please look at the list and post anything else you’d like to

 get included for 2.6.1.


 Thanks
 +Vinod


 On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli 

 vino...@hortonworks.commailto:vino...@hortonworks.com wrote:


 Alright, I’d like to make progress while the issue is hot.

 I created a label to discuss on the candidate list of patches:


 https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate
 
 https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate
 


 Next steps, I’ll do the following
   - Review 2.7 and 2.8 blocker/critical tickets and see what makes
 sense

 for 2.6.1 and add as candidates

   - I haven’t reviewed the current list yet, the seed list is from this

 email thread. Will review them.

   - I also have a bunch of patches that I’d like to include, will
 update

 them right away.


 Others, please look at the current list and let me know what else
 you’d

 like to include.


 I’d like to keep this ‘candidate-collection’ cycle’ for a max of a
 week

 and then start the release process. @Akira, let’s sync up offline on
 how to
 take this forward in terms of the release process.


 Thanks
 +Vinod

Re: Planning Hadoop 2.6.1 release

[Sangjin Lee]

Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
applies to 2.6.

Thanks,
Sangjin

On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli 
vino...@hortonworks.com wrote:

 I’ve added them all to the 2.6.1-candidate list. I included everything
 even though some of them are major tickets. The list is getting large, we
 will have to cut these down once we get down to the next phase of figuring
 out what to include and what not to.

 Thanks
 +Vinod

  On Jul 21, 2015, at 2:15 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp
 wrote:
 
  Thanks Vinod for updating the candidate list.
  I'd like to include the followings 12 JIRAs:
 
  * YARN-3641
  * YARN-3585
  * YARN-2910
  * HDFS-8431
  * HDFS-7830
  * HDFS-7763
  * HDFS-7742
  * HDFS-7235
  * HDFS-7225
  * MAPREDUCE-6324
  * HADOOP-11934
  * HADOOP-11491
 
  Thanks,
  Akira
 
  On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
   - I also have a bunch of patches that I’d like to include, will update
 them right away.
 
  I’ve just finished this. The latest 2.6.1-candidate list is up at 64
 JIRAs.
 
  Others, please look at the list and post anything else you’d like to
 get included for 2.6.1.
 
  Thanks
  +Vinod
 
 
  On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli 
 vino...@hortonworks.commailto:vino...@hortonworks.com wrote:
 
  Alright, I’d like to make progress while the issue is hot.
 
  I created a label to discuss on the candidate list of patches:
 https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate
 https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate
 
  Next steps, I’ll do the following
   - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense
 for 2.6.1 and add as candidates
   - I haven’t reviewed the current list yet, the seed list is from this
 email thread. Will review them.
   - I also have a bunch of patches that I’d like to include, will update
 them right away.
 
  Others, please look at the current list and let me know what else you’d
 like to include.
 
  I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week
 and then start the release process. @Akira, let’s sync up offline on how to
 take this forward in terms of the release process.
 
  Thanks
  +Vinod
 
 
 
 

[jira] [Created] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6372:
--

 Summary: clean up several issues with TimelineServicePerformance
 Key: MAPREDUCE-6372
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: YARN-2928
Reporter: Sangjin Lee
Assignee: Sangjin Lee


We found a few issues with the TimelineServicePerformanceV2 test driver while 
running it for the performance tests. Filing this JIRA to fix those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6293) uberized job fails with the job classloader enabled

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6293:
--

 Summary: uberized job fails with the job classloader enabled
 Key: MAPREDUCE-6293
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6293
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.6.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee


An uberized job fails if the job classloader is enabled and the job needs to 
use the thread context classloader to load a class. Some example error in the 
log:

{quote}
2015-03-23 23:28:34,675 INFO [main\] 
org.apache.hadoop.mapreduce.v2.util.MRApps: Creating job classloader
...
2015-03-23 23:28:42,096 ERROR [uber-SubtaskRunner\] 
cascading.provider.ServiceLoader: unable to find service class: 
cascading.tuple.hadoop.collect.HadoopTupleMapFactory, with exception: 
java.lang.ClassNotFoundException: 
cascading.tuple.hadoop.collect.HadoopTupleMapFactory
{quote}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Thinking ahead to hadoop-2.7

[Sangjin Lee]

Late January sounds fine to me. I think we should be able to wrap it up
much earlier than that (hopefully).

Thanks,
Sangjin

On Tue, Dec 2, 2014 at 5:19 PM, Arun C Murthy a...@hortonworks.com wrote:

 Sangjin/Karthik,

  How about planning on hadoop-2.8 by late Jan? Thoughts?

 thanks,
 Arun

 On Dec 2, 2014, at 11:09 AM, Sangjin Lee sjl...@gmail.com wrote:

  If 2.7 is being positioned as the JDK7-only release, then it would be
 good
  to know how 2.8 lines up in terms of timing. Our interest is landing the
  shared cache feature (YARN-1492)... Thanks.
 
  Sangjin
 
  On Mon, Dec 1, 2014 at 2:55 PM, Karthik Kambatla ka...@cloudera.com
 wrote:
 
  Thanks for starting this thread, Arun.
 
  Your proposal seems reasonable to me. I suppose we would like new
 features
  and improvements to go into 2.8 then? If yes, what time frame are we
  looking at for 2.8? Looking at YARN, it would be nice to get a release
 with
  shared-cache and a stable version of reservation work. I believe they
 are
  well under way and should be ready in a few weeks.
 
  Regarding 2.7 release specifics, do you plan to create a branch off of
  current branch-2.6 and update all issues marked fixed for 2.7 to be
 fixed
  for 2.8?
 
  Thanks
  Karthik
 
  On Mon, Dec 1, 2014 at 2:42 PM, Arun Murthy a...@hortonworks.com
 wrote:
 
  Folks,
 
  With hadoop-2.6 out it's time to think ahead.
 
  As we've discussed in the past, 2.6 was the last release which supports
  JDK6.
 
  I'm thinking it's best to try get 2.7 out in a few weeks (maybe by the
  holidays) with just the switch to JDK7 (HADOOP-10530) and possibly
  support for JDK-1.8 (as a runtime) via HADOOP-11090.
 
  This way we can start with the stable base of 2.6 and switch over to
  JDK7 to allow our downstream projects to use either for a short time
  (hadoop-2.6 or hadoop-2.7).
 
  I'll update the Roadmap wiki accordingly.
 
  Thoughts?
 
  thanks,
  Arun
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity
  to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby notified
  that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
  immediately
  and delete it from your system. Thank You.
 
 

 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/hdp/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: Guava

[Sangjin Lee]

FYI, we have an existing ApplicationClassLoader implementation that is used
to isolate client/task classes from the rest. If we're going down the route
of classloader isolation on this, it would be good to come up with a
coherent strategy regarding both of these.

As a more practical step, I like the idea of isolating usage of guava that
breaks with guava 16 and later. I assume (but I haven't looked into it)
that it's fairly straightforward to isolate them and fix them. That work
could be done at any time without any version upgrades or impacting users.

On Mon, Nov 10, 2014 at 9:26 AM, Alejandro Abdelnur tuc...@gmail.com
wrote:

 IMO we should:

 1* have a clean and thin client API JAR (which does not drag any 3rd party
 dependencies, or a well defined small set -i.e. slf4j  log4j-)
 2* have a client implementation that uses a classloader to isolate client
 impl 3rd party deps from app dependencies.

 #2 can be done using a stock URLClassLoader (i would just subclass it to
 forbid packages in the API JAR and exposed 3rd parties to be loaded from
 the app JAR)

 #1 is the tricky thing as our current API modules don't have a clean
 API/impl separation.

 thx
 PS: If folks are interested in pursing this, I can put together a prototype
 of how  #2 would work (I don't think it will be more than 200 lines of
 code)


 On Mon, Nov 10, 2014 at 5:18 AM, Steve Loughran ste...@hortonworks.com
 wrote:

  Yes, Guava is a constant pain; there's lots of open JIRAs related to it,
 as
  its the one we can't seamlessly upgrade. Not unless we do our own fork
 and
  reinsert the missing classes.
 
  The most common uses in the code are
 
  @VisibleForTesting (easily replicated)
  and the Precondition.check() operations
 
  The latter is also easily swapped out, and we could even add the check
 they
  forgot:
  Preconditions.checkArgNotNull(argname, arg)
 
 
  These are easy; its the more complex data structures that matter more.
 
  I think for Hadoop 2.7  java 7 we need to look at this problem and do
  something. Even if we continue to ship Guava 11 so that the HBase team
  don't send any (more) death threats, we can/should rework Hadoop to build
  and run against Guava 16+ too. That's needed to fix some of the recent
 java
  7/8+ changes.
 
  -Everything in v11 dropped from v16 MUST  to be implemented with our own
  versions.
  -anything tagged as deprecated in 11+ SHOULD be replaced by newer stuff,
  wherever possible.
 
  I think for 2.7+ we should add some new profiles to the POM, for Java 8
 and
  9 alongside the new baseline java 7. For those later versions we could
  perhaps mandate Guava 16.
 
 
 
  On 10 November 2014 00:42, Arun C Murthy a...@hortonworks.com wrote:
 
   ... has been a constant pain w.r.t compatibility etc.
  
   Should we consider adopting a policy to not use guava in
  Common/HDFS/YARN?
  
   MR doesn't matter too much since it's application-side issue, it does
  hurt
   end-users though since they still might want a newer guava-version, but
  at
   least they can modify MR.
  
   Thoughts?
  
   thanks,
   Arun
  
  
   --
   CONFIDENTIALITY NOTICE
   NOTICE: This message is intended for the use of the individual or
 entity
  to
   which it is addressed and may contain information that is confidential,
   privileged and exempt from disclosure under applicable law. If the
 reader
   of this message is not the intended recipient, you are hereby notified
  that
   any printing, copying, dissemination, distribution, disclosure or
   forwarding of this communication is strictly prohibited. If you have
   received this communication in error, please contact the sender
  immediately
   and delete it from your system. Thank You.
  
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 

[jira] [Created] (MAPREDUCE-6094) TestMRCJCFileInputFormat.testAddInputPath() fails on trunk

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6094:
--

 Summary: TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
 Key: MAPREDUCE-6094
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6094
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Sangjin Lee
Priority: Minor


{noformat}
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.624 sec  
FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat
testAddInputPath(org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat)
  Time elapsed: 0.886 sec   ERROR!
java.io.IOException: No FileSystem for scheme: s3
at 
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2583)
at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2590)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2629)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at 
org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat.testAddInputPath(TestMRCJCFileInputFormat.java:55)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6091) YARNRunner.getJobStatus() fails with ApplicationNotFoundException if the job rolled off the RM view

[Sangjin Lee (JIRA)]

Sangjin Lee created MAPREDUCE-6091:
--

 Summary: YARNRunner.getJobStatus() fails with 
ApplicationNotFoundException if the job rolled off the RM view
 Key: MAPREDUCE-6091
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6091
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 2.1.0-beta
Reporter: Sangjin Lee


If you query the job status of a job that rolled off the RM view via 
YARNRunner.getJobStatus(), it fails with an ApplicationNotFoundException. For 
example,

{noformat}
2014-09-15 07:09:51,084 ERROR org.apache.pig.tools.grunt.Grunt: ERROR 6017: 
JobID: job_1410289045532_90542 Reason: java.io.IOException: 
org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application 
with id 'application_1410289045532_90542' doesn't exist in RM.
at 
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:288)
at 
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:150)
at 
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:337)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2058)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2054)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1547)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2052)

at 
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:348)
at 
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:559)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1547)
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:599)
at 
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.checkRunningState(ControlledJob.java:257)
at 
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.checkState(ControlledJob.java:282)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.pig.backend.hadoop23.PigJobControl.checkState(PigJobControl.java:120)
at 
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:180)
at java.lang.Thread.run(Thread.java:662)
at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:279)
{noformat}

Prior to 2.1.0, it used to be able to fall back onto the job history server and 
get the status.

This appears to be introduced by YARN-873. YARN-873 changed ClientRMService to 
throw an ApplicationNotFoundException on an unknown app id (from returning 
null). But MR's ClientServiceDelegate was never modified to change its behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Migration from subversion to git for version control

[Sangjin Lee]

+1 (non-binding)


On Sat, Aug 9, 2014 at 9:34 AM, Sandy Ryza sandy.r...@cloudera.com wrote:

 +1 (binding)


 On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla ka...@cloudera.com
 wrote:

  I have put together this proposal based on recent discussion on this
 topic.
 
  Please vote on the proposal. The vote runs for 7 days.
 
 1. Migrate from subversion to git for version control.
 2. Force-push to be disabled on trunk and branch-* branches. Applying
 changes from any of trunk/branch-* to any of branch-* should be
 through
 git cherry-pick -x.
 3. Force-push on feature-branches is allowed. Before pulling in a
 feature, the feature-branch should be rebased on latest trunk and the
 changes applied to trunk through git rebase --onto or git
 cherry-pick
 commit-range.
 4. Every time a feature branch is rebased on trunk, a tag that
 identifies the state before the rebase needs to be created (e.g.
 tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted
 once
 the feature is pulled into trunk and the tags are no longer useful.
 5. The relevance/use of tags stay the same after the migration.
 
  Thanks
  Karthik
 
  PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of
  vote and will be Lazy 2/3 majority of PMC members.