Re: [DISCUSS] making Ozone a separate Apache project

2020-05-15 Thread Subru Krishnan 
+1.

Thanks,
Subru

On Thu, May 14, 2020 at 10:00 PM Akira Ajisaka  wrote:

> +1
>
> -Akira
>
> On Wed, May 13, 2020 at 4:53 PM Elek, Marton  wrote:
>
> >
> >
> > I would like to start a discussion to make a separate Apache project for
> > Ozone
> >
> >
> >
> > ### HISTORY [1]
> >
> >   * Apache Hadoop Ozone development started on a feature branch of
> > Hadoop repository (HDFS-7240)
> >
> >   * In the October of 2017 a discussion has been started to merge it to
> > the Hadoop main branch
> >
> >   * After a long discussion it's merged to Hadoop trunk at the March of
> > 2018
> >
> >   * During the discussion of the merge, it was suggested multiple times
> > to create a separated project for the Ozone. But at that time:
> >  1). Ozone was tightly integrated with Hadoop/HDFS
> >  2). There was an active plan to use Block layer of Ozone (HDDS or
> > HDSL at that time) as the block level of HDFS
> >  3). The community of Ozone was a subset of the HDFS community
> >
> >   * The first beta release of Ozone was just released. Seems to be a
> > good time before the first GA to make a decision about the future.
> >
> >
> >
> > ### WHAT HAS BEEN CHANGED
> >
> >   During the last years Ozone became more and more independent both at
> > the community and code side. The separation has been suggested again and
> > again (for example by Owen [2] and Vinod [3])
> >
> >
> >
> >   From COMMUNITY point of view:
> >
> >
> >* Fortunately more and more new contributors are helping Ozone.
> > Originally the Ozone community was a subset of HDFS project. But now a
> > bigger and bigger part of the community is related to Ozone only.
> >
> >* It seems to be easier to _build_ the community as a separated
> project.
> >
> >* A new, younger project might have different practices
> > (communication, commiter criteria, development style) compared to old,
> > mature project
> >
> >* It's easier to communicate (and improve) these standards in a
> > separated projects with clean boundaries
> >
> >* Separated project/brand can help to increase the adoption rate and
> > attract more individual contributor (AFAIK it has been seen in Submarine
> > after a similar move)
> >
> >   * Contribution process can be communicated more easily, we can make
> > first time contribution more easy
> >
> >
> >
> >   From CODE point of view Ozone became more and more independent:
> >
> >
> >   * Ozone has different release cycle
> >
> >   * Code is already separated from Hadoop code base
> > (apache/hadoop-ozone.git)
> >
> >   * It has separated CI (github actions)
> >
> >   * Ozone uses different (more strict) coding style (zero toleration of
> > unit test / checkstyle errors)
> >
> >   * The code itself became more and more independent from Hadoop on
> > Maven level. Originally it was compiled together with the in-tree latest
> > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC,
> > Configuration...)
> >
> >   * It starts to use multiple version of Hadoop (on client side)
> >
> >   * Volume of resolved issues are already very high on Ozone side (Ozone
> > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all
> > together in the last 2-3 months)
> >
> >
> > Summary: Before the first Ozone GA release, It seems to be a good time
> > to discuss the long-term future of Ozone. Managing it as a separated TLP
> > project seems to have more benefits.
> >
> >
> > Please let me know what your opinion is...
> >
> > Thanks a lot,
> > Marton
> >
> >
> >
> >
> >
> > [1]: For more details, see:
> > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md
> >
> > [2]:
> >
> >
> https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E
> >
> > [3]:
> >
> >
> https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
>

Re: [VOTE] EOL Hadoop branch-2.8

2020-03-03 Thread Subru Krishnan 
+1

On Tue, Mar 3, 2020 at 11:43 AM Wangda Tan  wrote:

> +1
>
> On Mon, Mar 2, 2020 at 8:50 PM Akira Ajisaka  wrote:
>
> > +1
> >
> > -Akira
> >
> > On Tue, Mar 3, 2020 at 4:55 AM Ayush Saxena  wrote:
> >
> > > +1 for marking 2.8 EOL
> > >
> > > -Ayush
> > >
> > > > On 03-Mar-2020, at 12:18 AM, Wei-Chiu Chuang 
> > wrote:
> > > >
> > > > I am sorry I forgot to start a VOTE thread.
> > > >
> > > > This is the "official" vote thread to mark branch-2.8 End of Life.
> This
> > > is
> > > > based on the following thread and the tracking jira (HADOOP-16880
> > > > ).
> > > >
> > > > This vote will run for 7 days and conclude on March 9th (Mon) 11am
> PST.
> > > >
> > > > Please feel free to share your thoughts.
> > > >
> > > > Thanks,
> > > > Weichiu
> > > >
> > > >> On Mon, Feb 24, 2020 at 10:28 AM Wei-Chiu Chuang <
> > weic...@cloudera.com>
> > > >> wrote:
> > > >>
> > > >> Looking at the EOL policy wiki:
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
> > > >>
> > > >> The Hadoop community can still elect to make security update for
> > EOL'ed
> > > >> releases.
> > > >>
> > > >> I think the EOL is to give more clarity to downstream applications
> > (such
> > > >> as HBase) the guidance of which Hadoop release lines are still
> active.
> > > >> Additionally, I don't think it is sustainable to maintain 6
> concurrent
> > > >> release lines in this big project, which is why I wanted to start
> this
> > > >> discussion.
> > > >>
> > > >> Thoughts?
> > > >>
> > > >>> On Mon, Feb 24, 2020 at 10:22 AM Sunil Govindan  >
> > > wrote:
> > > >>>
> > > >>> Hi Wei-Chiu
> > > >>>
> > > >>> Extremely sorry for the late reply here.
> > > >>> Cud u pls help to add more clarity on defining what will happen for
> > > >>> branch-2.8 when we call EOL.
> > > >>> Does this mean that, no more release coming out from this branch,
> or
> > > some
> > > >>> more additional guidelines?
> > > >>>
> > > >>> - Sunil
> > > >>>
> > > >>>
> > > >>> On Mon, Feb 24, 2020 at 11:47 PM Wei-Chiu Chuang
> > > >>>  wrote:
> > > >>>
> > >  This thread has been running for 7 days and no -1.
> > > 
> > >  Don't think we've established a formal EOL process, but to
> publicize
> > > the
> > >  EOL, I am going to file a jira, update the wiki and post the
> > > >>> announcement
> > >  to general@ and user@
> > > 
> > >  On Wed, Feb 19, 2020 at 1:40 PM Dinesh Chitlangia <
> > > >>> dineshc@gmail.com>
> > >  wrote:
> > > 
> > > > Thanks Wei-Chiu for initiating this.
> > > >
> > > > +1 for 2.8 EOL.
> > > >
> > > > On Tue, Feb 18, 2020 at 10:48 PM Akira Ajisaka <
> > aajis...@apache.org>
> > > > wrote:
> > > >
> > > >> Thanks Wei-Chiu for starting the discussion,
> > > >>
> > > >> +1 for the EoL.
> > > >>
> > > >> -Akira
> > > >>
> > > >> On Tue, Feb 18, 2020 at 4:59 PM Ayush Saxena <
> ayush...@gmail.com>
> > >  wrote:
> > > >>
> > > >>> Thanx Wei-Chiu for initiating this
> > > >>> +1 for marking 2.8 EOL
> > > >>>
> > > >>> -Ayush
> > > >>>
> > >  On 17-Feb-2020, at 11:14 PM, Wei-Chiu Chuang <
> > > >>> weic...@apache.org>
> > > >> wrote:
> > > 
> > >  The last Hadoop 2.8.x release, 2.8.5, was GA on September
> 15th
> > >  2018.
> > > 
> > >  It's been 17 months since the release and the community by and
> > >  large
> > > >> have
> > >  moved up to 2.9/2.10/3.x.
> > > 
> > >  With Hadoop 3.3.0 over the horizon, is it time to start the
> EOL
> > > >>> discussion
> > >  and reduce the number of active branches?
> > > >>>
> > > >>>
> > > >>>
> -
> > > >>> To unsubscribe, e-mail:
> common-dev-unsubscr...@hadoop.apache.org
> > > >>> For additional commands, e-mail:
> > > >>> common-dev-h...@hadoop.apache.org
> > > >>>
> > > >>>
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > >
> > > -
> > > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] Remove Ozone and Submarine from Hadoop repo

2019-10-24 Thread Subru Krishnan 
+1.

Thanks,
Subru

On Thu, Oct 24, 2019 at 12:51 AM 张铎(Duo Zhang) 
wrote:

> +1
>
> Akira Ajisaka  于2019年10月24日周四 下午3:21写道：
>
> > Hi folks,
> >
> > Both Ozone and Apache Submarine have separate repositories.
> > Can we remove these modules from hadoop-trunk?
> >
> > Regards,
> > Akira
> >
>

Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree

2019-09-17 Thread Subru Krishnan 
+1 (binding).

IIUC, there will not be an Ozone module in trunk anymore as that was my
only concern from the original discussion thread? IMHO, this should be the
default approach for new modules.

On Tue, Sep 17, 2019 at 9:58 AM Salvatore LaMendola (BLOOMBERG/ 731 LEX) <
slamendo...@bloomberg.net> wrote:

> +1
>
> From: e...@apache.org At: 09/17/19 05:48:32To:  hdfs-dev@hadoop.apache.org,
> mapreduce-...@hadoop.apache.org,  common-...@hadoop.apache.org,
> yarn-...@hadoop.apache.org
> Subject: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk
> source tree
>
>
> TLDR; I propose to move Ozone related code out from Hadoop trunk and
> store it in a separated *Hadoop* git repository apache/hadoop-ozone.git
>
>
> When Ozone was adopted as a new Hadoop subproject it was proposed[1] to
> be part of the source tree but with separated release cadence, mainly
> because it had the hadoop-trunk/SNAPSHOT as compile time dependency.
>
> During the last Ozone releases this dependency is removed to provide
> more stable releases. Instead of using the latest trunk/SNAPSHOT build
> from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now).
>
> As we have no more strict dependency between Hadoop trunk SNAPSHOT and
> Ozone trunk I propose to separate the two code base from each other with
> creating a new Hadoop git repository (apache/hadoop-ozone.git):
>
> With moving Ozone to a separated git repository:
>
>   * It would be easier to contribute and understand the build (as of now
> we always need `-f pom.ozone.xml` as a Maven parameter)
>   * It would be possible to adjust build process without breaking
> Hadoop/Ozone builds.
>   * It would be possible to use different Readme/.asf.yaml/github
> template for the Hadoop Ozone and core Hadoop. (For example the current
> github template [2] has a link to the contribution guideline [3]. Ozone
> has an extended version [4] from this guideline with additional
> information.)
>   * Testing would be more safe as it won't be possible to change core
> Hadoop and Hadoop Ozone in the same patch.
>   * It would be easier to cut branches for Hadoop releases (based on the
> original consensus, Ozone should be removed from all the release
> branches after creating relase branches from trunk)
>
>
> What do you think?
>
> Thanks,
> Marton
>
> [1]:
>
> https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8
> c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E
> [2]:
>
> https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md
> [3]: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
> [4]:
>
> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

2019-08-21 Thread Subru Krishnan 
+1

On Wed, Aug 21, 2019 at 8:52 AM Kihwal Lee 
wrote:

> +1
>
> Kihwal
>
> On Tue, Aug 20, 2019 at 10:03 PM Wangda Tan  wrote:
>
> > Hi all,
> >
> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> > and 3.0 EOL. This is based on discussions of [1]
> >
> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Wangda
> >
> > [1]
> >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> > ,
> >
>

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

2019-02-07 Thread Subru Krishnan 
Belated binding +1 from my side (on vacation).

Thanks Jonathan for initiating this.

-Subru

On Thu, Feb 7, 2019 at 6:13 PM Jonathan Hung  wrote:

> My non-binding +1 to finish. This vote passes with 6 binding +1, 3
> non-binding +1, and no vetoes. We will make the changes as part
> of HADOOP-15711, please follow there.
>
> Thanks all!
>
> Jonathan Hung
>
>
> On Tue, Feb 5, 2019 at 11:38 PM Akira Ajisaka  wrote:
>
> > +1
> >
> > -Akira
> >
> > On Wed, Feb 6, 2019 at 9:13 AM Wangda Tan  wrote:
> > >
> > > +1, make sense to me.
> > >
> > > On Tue, Feb 5, 2019 at 3:29 PM Konstantin Shvachko <
> shv.had...@gmail.com
> > >
> > > wrote:
> > >
> > > > +1 Makes sense to me.
> > > >
> > > > Thanks,
> > > > --Konst
> > > >
> > > > On Mon, Feb 4, 2019 at 6:14 PM Jonathan Hung 
> > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > Starting a vote based on the discuss thread [1] for moving branch-2
> > > > > precommit/nightly test builds to openjdk8. After this change, the
> > test
> > > > > phase for precommit builds [2] and branch-2 nightly build [3] will
> > run on
> > > > > openjdk8. To maintain source compatibility, these builds will still
> > run
> > > > > their compile phase for branch-2 on openjdk7 as they do now (in
> > addition
> > > > to
> > > > > compiling on openjdk8).
> > > > >
> > > > > Vote will run for three business days until Thursday Feb 7 6:00PM
> > PDT.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> >
> https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
> > > > >
> > > > > [2]
> > > > >
> > > >
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> > > > >
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> > > > >
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
> > > > >
> > > > >
> > > >
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
> > > > >
> > > > > [3]
> > > > >
> > > > >
> > > >
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> > > > >
> > > > > Jonathan Hung
> > > > >
> > > >
> >
>

Re: [DISCUSS] Move to gitbox

2018-12-11 Thread Subru Krishnan 
+1.

On Tue, Dec 11, 2018 at 12:10 AM Mukul Kumar Singh 
wrote:

> +1
> -Mukul
>
> On 12/11/18, 9:56 AM, "Weiwei Yang"  wrote:
>
> +1
>
> On Tue, Dec 11, 2018 at 10:51 AM Anu Engineer <
> aengin...@hortonworks.com>
> wrote:
>
> > +1
> > --Anu
> >
> >
> > On 12/10/18, 6:38 PM, "Vinayakumar B" 
> wrote:
> >
> > +1
> >
> > -Vinay
> >
> > On Mon, 10 Dec 2018, 1:22 pm Elek, Marton  wrote:
> >
> > >
> > > Thanks Akira,
> > >
> > > +1 (non-binding)
> > >
> > > I think it's better to do it now at a planned date.
> > >
> > > If I understood well the only bigger task here is to update
> all the
> > > jenkins jobs. (I am happy to help/contribute what I can do)
> > >
> > >
> > > Marton
> > >
> > > On 12/8/18 6:25 AM, Akira Ajisaka wrote:
> > > > Hi all,
> > > >
> > > > Apache Hadoop git repository is in git-wip-us server and it
> will be
> > > > decommissioned.
> > > > If there are no objection, I'll file a JIRA ticket with
> INFRA to
> > > > migrate to https://gitbox.apache.org/ and update
> documentation.
> > > >
> > > > According to ASF infra team, the timeframe is as follows:
> > > >
> > > >> - December 9th 2018 -> January 9th 2019: Voluntary
> (coordinated)
> > > relocation
> > > >> - January 9th -> February 6th: Mandated (coordinated)
> relocation
> > > >> - February 7th: All remaining repositories are mass
> migrated.
> > > >> This timeline may change to accommodate various scenarios.
> > > >
> > > > If we got consensus by January 9th, I can file a ticket with
> INFRA
> > and
> > > > migrate it.
> > > > Even if we cannot got consensus, the repository will be
> migrated by
> > > > February 7th.
> > > >
> > > > Regards,
> > > > Akira
> > > >
> > > >
> > -
> > > > To unsubscribe, e-mail:
> yarn-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail:
> yarn-dev-h...@hadoop.apache.org
> > > >
> > >
> > >
> -
> > > To unsubscribe, e-mail:
> common-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail:
> common-dev-h...@hadoop.apache.org
> > >
> > >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
>
>
> --
> Weiwei Yang
>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>

New committer: Botong Huang

2018-11-21 Thread Subru Krishnan 
The Project Management Committee (PMC) for Apache Hadoophas invited
Botong Huang to become a committer and we are pleased to announce that
he has accepted.
Being a committer enables easier contribution to theproject since
there is no need to go via the patchsubmission process. This should
enable better productivity.Being a PMC member enables assistance with
the managementand to guide the direction of the project.

Congrats and welcome aboard.

-Subru

Re: Hadoop 3.2 Release Plan proposal

2018-07-19 Thread Subru Krishnan 
Thanks Sunil for volunteering to lead the release effort. I am generally
supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we already
have too many branches to be maintained. I already see many commits are in
different branches with no apparent rationale, for e.g: 3.1 has commits
which are absent in 3.0 etc.

Additionally AFAIK 3.x has not been deployed in any major production
setting so the cost of adding features should be minimal.

Thoughts?

-Subru

On Thu, Jul 19, 2018 at 12:31 AM, Sunil G  wrote:

> Thanks Steve, Aaron, Wangda for sharing thoughts.
>
> Yes, important changes and features are much needed, hence we will be
> keeping the door open for them as possible. Also considering few more
> offline requests from other folks, I think extending the timeframe by
> couple of weeks makes sense (including a second RC buffer) and this should
> ideally help us to ship this by September itself.
>
> Revised dates (I will be updating same in Roadmap wiki as well)
>
> - Feature freeze date : all features to merge by August 21, 2018.
>
> - Code freeze date : blockers/critical only, no improvements and non
> blocker/critical
>
> bug-fixes  August 31, 2018.
>
> - Release date: September 15, 2018
>
> Thank Eric and Zian, I think Wangda has already answered your questions.
>
> Thanks
> Sunil
>
>
> On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan  wrote:
>
> > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
> >
> > To concerns from Steve,
> >
> > It is a good idea to keep the door open to get important changes /
> > features in before cutoff. I would prefer to keep the proposed release
> date
> > to make sure things can happen earlier instead of last minute and we all
> > know that releases are always get delayed :). I'm also fine if we want
> get
> > another several weeks time.
> >
> > Regarding of 3.3 release, I would suggest doing that before thanksgiving.
> > Do you think is it good or too early / late?
> >
> > Eric,
> >
> > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get merged
> > in time, we probably not need the YARN-8220.
> >
> > Sunil,
> >
> > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap
> > with proposed plan as well? We can fill feature list first before getting
> > consensus of time.
> >
> > Thanks,
> > Wangda
> >
> > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri  >
> > wrote:
> >
> >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran 
> >> wrote:
> >>
> >> >
> >> >
> >> > On 16 Jul 2018, at 23:45, Sunil G  >> > sun...@apache.org>> wrote:
> >> >
> >> > I would also would like to take this opportunity to come up with a
> >> detailed
> >> > plan.
> >> >
> >> > - Feature freeze date : all features should be merged by August 10,
> >> 2018.
> >> >
> >> >
> >> >
> >> > 
> >>
> >> >
> >> > Please let me know if I missed any features targeted to 3.2 per this
> >> >
> >> >
> >> > Well there these big todo lists for S3 & S3Guard.
> >> >
> >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >> >
> >> >
> >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
> >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >> >
> >> > I don't think this is quite ready yet, I've been doing work on it, but
> >> if
> >> > we have a 3 week deadline, I'm going to expect some timely reviews on
> >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >> >
> >> > I've uprated that to a blocker feature; will review the S3 & S3Guard
> >> JIRAs
> >> > to see which of those are blocking. Then there are some pressing
> "guave,
> >> > java 9 prep"
> >> >
> >> >
> >>  I can help with this part if you like.
> >>
> >>
> >>
> >> >
> >> >
> >> >
> >> > timeline. I would like to volunteer myself as release manager of 3.2.0
> >> > release.
> >> >
> >> >
> >> > well volunteered!
> >> >
> >> >
> >> >
> >> Yes, thank you for stepping up.
> >>
> >>
> >> >
> >> > I think this raises a good q: what timetable should we have for the
> >> 3.2. &
> >> > 3.3 releases; if we do want a faster cadence, then having the outline
> >> time
> >> > from the 3.2 to the 3.3 release means that there's less concern about
> >> > things not making the 3.2 dealine
> >> >
> >> > -Steve
> >> >
> >> >
> >> Good idea to mitigate the short deadline.
> >>
> >> -AF
> >>
> >
>

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

2018-07-06 Thread Subru Krishnan 
t;
> >
> >
> > Thanks
> > Anu
> >
> >
> > On 7/6/18, 10:24 AM, "Arpit Agarwal"  mailto:
> >
> > aagar...@hortonworks.com>> wrote:
> >
> >
> >   -1 for the force push. Nothing is broken in trunk. The history
> looks
> >
> > ugly for two commits and we can live with it.
> >
> >
> >   The revert restored the branch to Giovanni's intent. i.e. only
> >
> > YARN-8435 is applied. Verified there is no delta between hashes
> 0d9804d and
> > 39ad989 (HEAD).
> >
> >
> >   39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge
> branch
> >
> > 't...
> >
> >   c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of
> >
> > https://git-...
> >
> >   99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to
> >
> > veri...
> >
> >   1726247 2018-07-05 haibochen@ │ o YARN-7556. Fair scheduler
> >
> > configurat...
> >
> >   0d9804d 2018-07-05 gifuma@apa o │ YARN-8435. Fix NPE when the same
> >
> > cli...
> >
> >   71df8c2 2018-07-05 nanda@apac o─┘ HDDS-212. Introduce
> >
> > NodeStateManager...
> >
> >
> >   Regards,
> >   Arpit
> >
> >
> >   On 7/5/18, 2:37 PM, "Subru Krishnan"  >
> > su...@apache.org>> wrote:
> >
> >
> >   Folks,
> >
> >   There was a merge commit accidentally pushed to trunk, you can
> >
> > find the
> >
> >   details in the mail thread [1].
> >
> >   I have raised an INFRA ticket [2] to reset/force push to clean
> up
> >
> > trunk.
> >
> >
> >   Can we have a quick vote for INFRA sign-off to proceed as this
> is
> >
> > blocking
> >
> >   all commits?
> >
> >   Thanks,
> >   Subru
> >
> >   [1]
> >
> >
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/
> 201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%
> 40mail.gmail.com%3E
> >
> >   [2] https://issues.apache.org/jira/browse/INFRA-16727
> >
> >
> >
> >   
> -
> >   To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >
> > <mailto:common-dev-unsubscr...@hadoop.apache.org
> > >
> >
> >   For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> > <mailto:common-dev-h...@hadoop.apache.org
> > >
> >
> >
> >
> >
> > 
> -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>  >
> > common-dev-unsubscr...@hadoop.apache.org>
> >
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> > <mailto:common-dev-h...@hadoop.apache.org
> > >
> >
> > 
> -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>  > yarn-dev-unsubscr...@hadoop.apache.org>
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>  > yarn-dev-h...@hadoop.apache.org>
> >
> >
> >
>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

2018-07-05 Thread Subru Krishnan 
Unfortunately since it was a merge commit, less straightforward to revert.
You can find the details in the original mail thread:
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%40mail.gmail.com%3E

On Thu, Jul 5, 2018 at 2:49 PM, Wei-Chiu Chuang  wrote:

> I'm sorry I come to this thread late.
> Anu commented on INFRA-16727 saying he reverted the commit. Do we still
> need the vote?
>
> Thanks
>
> On Thu, Jul 5, 2018 at 2:47 PM Rohith Sharma K S <
> rohithsharm...@apache.org> wrote:
>
>> +1
>>
>> On 5 July 2018 at 14:37, Subru Krishnan  wrote:
>>
>> > Folks,
>> >
>> > There was a merge commit accidentally pushed to trunk, you can find the
>> > details in the mail thread [1].
>> >
>> > I have raised an INFRA ticket [2] to reset/force push to clean up trunk.
>> >
>> > Can we have a quick vote for INFRA sign-off to proceed as this is
>> blocking
>> > all commits?
>> >
>> > Thanks,
>> > Subru
>> >
>> > [1]
>> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%
>> > 3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%
>> > 40mail.gmail.com%3E
>> > [2] https://issues.apache.org/jira/browse/INFRA-16727
>> >
>>
>> --
>> A very happy Hadoop contributor
>>
>

[VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

2018-07-05 Thread Subru Krishnan 
Folks,

There was a merge commit accidentally pushed to trunk, you can find the
details in the mail thread [1].

I have raised an INFRA ticket [2] to reset/force push to clean up trunk.

Can we have a quick vote for INFRA sign-off to proceed as this is blocking
all commits?

Thanks,
Subru

[1]
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%40mail.gmail.com%3E
[2] https://issues.apache.org/jira/browse/INFRA-16727

[DISCUSS] 2.9+ stabilization branch

2018-02-26 Thread Subru Krishnan 
Folks,

We (i.e. Microsoft) have started stabilization of 2.9 for our production
deployment. During planning, we realized that we need to backport 3.x
features to support GPUs (and more resource types like network IO) natively
as part of the upgrade. We'd like to share that work with the community.

Instead of stabilizing the base release and cherry-picking fixes back to
Apache, we want to work publicly and push fixes directly into
trunk/.../branch-2 for a stable 2.10.0 release. Our goal is to create a
bridge release for our production clusters to the 3.x series and to address
scalability problems in large clusters (N*10k nodes). As we find issues, we
will file JIRAs and track resolution of significant regressions/faults in
wiki. Moreover, LinkedIn also has committed plans for a production
deployment of the same branch. We welcome broad participation, particularly
since we'll be stabilizing relatively new features.

The exact list of features we would like to backport in YARN are:

   - Support for Resource types [1][2]
   - Native support for GPUs[3]
   - Absolute Resource configuration in CapacityScheduler [4]


With regards to HDFS, we are currently looking at mainly fixes to Router
based Federation and Windows specific fixes which should anyways flow
normally.

Thoughts?

Thanks,
Subru/Arun

[1] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27786.html
[2] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28281.html
[3] https://issues.apache.org/jira/browse/YARN-6223
[4] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28772.html

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

2017-11-17 Thread Subru Krishnan 
Wrapping up the vote with my +1.

Deployed RC3 on a federated YARN cluster with 6 sub-clusters:
- ran multiple sample jobs
- enabled opportunistic containers and submitted more samples
- configured HDFS federation and reran jobs.


With 13 binding +1s and 7 non-binding +1s and no +/-1s, pleased to announce
the vote is passed successfully.

Thanks to the many of you who contributed to the release and made this
possible and to everyone in this thread who took the time/effort to
validate and vote!

We’ll push the release bits and send out an announcement for 2.9.0 soon.

Cheers,
Subru

On Fri, Nov 17, 2017 at 12:41 PM Eric Payne 
wrote:

> Thanks Arun and Subru for the hard work on this release.
>
> +1 (binding)
>
> Built from source and stood up a pseudo cluster with 4 NMs
>
> Tested the following:
>
> o User limits are honored during In-queue preemption
>
> o Priorities are honored during In-queue preemption
>
> o Can kill applications from the command line
>
> o Users with different weights are assigned resources proportional to
> their weights.
>
> Thanks,
> -Eric Payne
>
>
> --
> *From:* Arun Suresh 
> *To:* yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; Hadoop
> Common ; Hdfs-dev <
> hdfs-dev@hadoop.apache.org>
> *Cc:* Subramaniam Krishnan 
> *Sent:* Monday, November 13, 2017 6:10 PM
>
> *Subject:* [VOTE] Release Apache Hadoop 2.9.0 (RC3)
>
> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
>
> More information about the 2.9.0 release plan can be found here:
> *
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> <
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> >*
>
> New RC is available at: *
> https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
> *
>
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
>
> The maven artifacts are available via repository.apache.org at:
> <
> https://www.google.com/url?q=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
> >*https://repository.apache.org/content/repositories/orgapachehadoop-1068/
>  >*
>
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
>
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
>
> Thanks,
> -Arun/Subru
>
>
>

[VOTE] Release Apache Hadoop 2.9.0 (RC2)

2017-11-12 Thread Subru Krishnan 
Hi Folks,

Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
2.8.2.

More information about the 2.9.0 release plan can be found here:
*https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
*

New RC is available at: http://home.apache.org/~asuresh/hadoop-2.9.0-RC2/


The RC tag in git is: release-2.9.0-RC2, and the latest commit id is:
1eb05c1dd48fbc9e4b375a76f2046a59103bbeb1.

The maven artifacts are available via repository.apache.org at:
https://repository.apache.org/content/repositories/orgapachehadoop-1067/


Please try the release and vote; the vote will run for the usual 5 days,
ending on Friday 17th November 2017 2pm PT time.

We want to give a big shout out to Sunil, Varun, Rohith, Wangda, Vrushali
and Inigo for the extensive testing/validation which helped prepare for
RC2. Do report your results in this vote as it'll be very useful to the
entire community.

Thanks,
-Subru/Arun

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC1)

2017-11-10 Thread Subru Krishnan 
Thanks John and Mukul for testing/voting on the RC but unfortunately we
have to cancel RC1 as we missed cherry-picking an addendum patch, courtesy
Rohith for pointing that out. Sincerely apologize, we'll be updating and
sending out RC2 asap.

-Subru/Arun

On Fri, Nov 10, 2017 at 2:35 AM, Mukul Kumar Singh <msi...@hortonworks.com>
wrote:

> +1 (non-binding)
>
> I built from source on Mac OS X 10.12.6 Java 1.8.0_111
>
> - Deployed on a single node cluster.
> - Deployed a ViewFS cluster with two hdfs mount points.
> - Performed basic sanity checks.
> - Performed basic DFS operations.
>
> Thanks,
> Mukul
>
>
>
>
>
>
> On 10/11/17, 7:09 AM, "Subru Krishnan" <su...@apache.org> wrote:
>
> >Hi Folks,
> >
> >Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be
> the
> >starting release for Apache Hadoop 2.9.x line - it includes 30 New
> Features
> >with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> >2.8.2 .
> >
> >More information about the 2.9.0 release plan can be found here:
> >*https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
> ><https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9>*
> >
> >New RC is available at: http://home.apache.org/~asuresh/hadoop-2.9.0-RC1/
> ><http://www.google.com/url?q=http%3A%2F%2Fhome.apache.org%
> 2F~asuresh%2Fhadoop-2.9.0-RC1%2F=D=1=
> AFQjCNE7BF35IDIMZID3hPqiNglWEVsTpg>
> >
> >The RC tag in git is: release-2.9.0-RC1, and the latest commit id is:
> >7d2ba3e8dd74d2631c51ce6790d59e50eeb7a846.
> >
> >The maven artifacts are available via repository.apache.org at:
> >https://repository.apache.org/content/repositories/orgapachehadoop-1066
> ><https://www.google.com/url?q=https%3A%2F%2Frepository.
> apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D&
> sntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg>
> >
> >Please try the release and vote; the vote will run for the usual 5 days,
> >ending on Tuesday 14th November 2017 6pm PST time.
> >
> >Thanks,
> >-Subru/Arun
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[VOTE] Release Apache Hadoop 2.9.0 (RC1)

2017-11-09 Thread Subru Krishnan 
Hi Folks,

Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
2.8.2 .

More information about the 2.9.0 release plan can be found here:
*https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
*

New RC is available at: http://home.apache.org/~asuresh/hadoop-2.9.0-RC1/


The RC tag in git is: release-2.9.0-RC1, and the latest commit id is:
7d2ba3e8dd74d2631c51ce6790d59e50eeb7a846.

The maven artifacts are available via repository.apache.org at:
https://repository.apache.org/content/repositories/orgapachehadoop-1066


Please try the release and vote; the vote will run for the usual 5 days,
ending on Tuesday 14th November 2017 6pm PST time.

Thanks,
-Subru/Arun

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

2017-11-09 Thread Subru Krishnan 
Thanks Vinod for your feedback, we'll incorporate it when we spin RC1.

-Subru/Arun

On Wed, Nov 8, 2017 at 5:34 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
wrote:

> A related point - I thought I mentioned this in one of the release
> preparation threads, but in any case.
>
> Starting 2.7.0, for every .0 release, we've been adding a disclaimer (to
> the voting thread as well as the final release) that the first release can
> potentially go through additional fixes to incompatible changes (besides
> stabilization fixes). We should do this with 2.9.0 too.
>
> This has some history - long before this, we tried two different things:
> (a) downstream projects consume an RC (b) downstream projects consume a
> release. Option (a) was tried many times but it was increasingly getting
> hard to manage this across all the projects that depend on Hadoop. When we
> tried option (b), we used to make .0 as a GA release, but downstream
> projects like Tez, Hive, Spark would come back and find an incompatible
> change - and now we were forced into a conundrum - is fixing this
> incompatible change itself an incompatibility? So to avoid this problem,
> we've started marking the first few releases as alpha eventually making a
> stable point release. Clearly, specific users can still use this in
> production as long as we the Hadoop community reserve the right to fix
> incompatibilities.
>
> Long story short, I'd just add to your voting thread and release notes
> that 2.9.0 still needs to be tested downstream and so users may want to
> wait for subsequent point releases.
>
> Thanks
> +Vinod
>
> > On Nov 8, 2017, at 12:43 AM, Subru Krishnan <su...@apache.org> wrote:
> >
> > We are canceling the RC due to the issue that Rohith/Sunil identified.
> The
> > issue was difficult to track down as it only happens when you use IP for
> ZK
> > (works fine with host names) and moreover if ZK and RM are co-located on
> > same machine. We are hopeful to get the fix in tomorrow and roll out RC1.
> >
> > Thanks to everyone for the extensive testing/validation. Hopefully cost
> to
> > replicate with RC1 is much lower.
> >
> > -Subru/Arun.
> >
> > On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos <
> kkarana...@gmail.com
> >> wrote:
> >
> >> +1 from me too.
> >>
> >> Did the following:
> >> 1) set up a 9-node cluster;
> >> 2) ran some Gridmix jobs;
> >> 3) ran (2) after enabling opportunistic containers (used a mix of
> >> guaranteed and opportunistic containers for each job);
> >> 4) ran (3) but this time enabling distributed scheduling of
> opportunistic
> >> containers.
> >>
> >> All the above worked with no issues.
> >>
> >> Thanks for all the effort guys!
> >>
> >> Konstantinos
> >>
> >>
> >>
> >> Konstantinos
> >>
> >> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger <ebad...@oath.com.invalid>
> >> wrote:
> >>
> >>> +1 (non-binding) pending the issue that Sunil/Rohith pointed out
> >>>
> >>> - Verified all hashes and checksums
> >>> - Built from source on macOS 10.12.6, Java 1.8.0u65
> >>> - Deployed a pseudo cluster
> >>> - Ran some example jobs
> >>>
> >>> Thanks,
> >>>
> >>> Eric
> >>>
> >>> On Tue, Nov 7, 2017 at 4:03 PM, Wangda Tan <wheele...@gmail.com>
> wrote:
> >>>
> >>>> Sunil / Rohith,
> >>>>
> >>>> Could you check if your configs are same as Jonathan posted configs?
> >>>> https://issues.apache.org/jira/browse/YARN-7453?
> >>> focusedCommentId=16242693&
> >>>> page=com.atlassian.jira.plugin.system.issuetabpanels:
> >>>> comment-tabpanel#comment-16242693
> >>>>
> >>>> And could you try if using Jonathan's configs can still reproduce the
> >>>> issue?
> >>>>
> >>>> Thanks,
> >>>> Wangda
> >>>>
> >>>>
> >>>> On Tue, Nov 7, 2017 at 1:52 PM, Arun Suresh <asur...@apache.org>
> >> wrote:
> >>>>
> >>>>> Thanks for testing Rohith and Sunil
> >>>>>
> >>>>> Can you please confirm if it is not a config issue at your end ?
> >>>>> We (both Jonathan and myself) just tried testing this on a fresh
> >>> cluster
> >>>>> (both automatic and manual) and we are not able to reproduce this.
> >

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

2017-11-08 Thread Subru Krishnan 
We are canceling the RC due to the issue that Rohith/Sunil identified. The
issue was difficult to track down as it only happens when you use IP for ZK
(works fine with host names) and moreover if ZK and RM are co-located on
same machine. We are hopeful to get the fix in tomorrow and roll out RC1.

Thanks to everyone for the extensive testing/validation. Hopefully cost to
replicate with RC1 is much lower.

-Subru/Arun.

On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos  wrote:

> +1 from me too.
>
> Did the following:
> 1) set up a 9-node cluster;
> 2) ran some Gridmix jobs;
> 3) ran (2) after enabling opportunistic containers (used a mix of
> guaranteed and opportunistic containers for each job);
> 4) ran (3) but this time enabling distributed scheduling of opportunistic
> containers.
>
> All the above worked with no issues.
>
> Thanks for all the effort guys!
>
> Konstantinos
>
>
>
> Konstantinos
>
> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger 
> wrote:
>
> > +1 (non-binding) pending the issue that Sunil/Rohith pointed out
> >
> > - Verified all hashes and checksums
> > - Built from source on macOS 10.12.6, Java 1.8.0u65
> > - Deployed a pseudo cluster
> > - Ran some example jobs
> >
> > Thanks,
> >
> > Eric
> >
> > On Tue, Nov 7, 2017 at 4:03 PM, Wangda Tan  wrote:
> >
> > > Sunil / Rohith,
> > >
> > > Could you check if your configs are same as Jonathan posted configs?
> > > https://issues.apache.org/jira/browse/YARN-7453?
> > focusedCommentId=16242693&
> > > page=com.atlassian.jira.plugin.system.issuetabpanels:
> > > comment-tabpanel#comment-16242693
> > >
> > > And could you try if using Jonathan's configs can still reproduce the
> > > issue?
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Tue, Nov 7, 2017 at 1:52 PM, Arun Suresh 
> wrote:
> > >
> > > > Thanks for testing Rohith and Sunil
> > > >
> > > > Can you please confirm if it is not a config issue at your end ?
> > > > We (both Jonathan and myself) just tried testing this on a fresh
> > cluster
> > > > (both automatic and manual) and we are not able to reproduce this.
> I've
> > > > updated the YARN-7453  jira/browse/YARN-7453
> > >
> > > > JIRA
> > > > with details of testing.
> > > >
> > > > Cheers
> > > > -Arun/Subru
> > > >
> > > > On Tue, Nov 7, 2017 at 3:17 AM, Rohith Sharma K S <
> > > > rohithsharm...@apache.org
> > > > > wrote:
> > > >
> > > > > Thanks Sunil for confirmation. Btw, I have raised YARN-7453
> > > > >  JIRA to track
> this
> > > > > issue.
> > > > >
> > > > > - Rohith Sharma K S
> > > > >
> > > > > On 7 November 2017 at 16:44, Sunil G  wrote:
> > > > >
> > > > >> Hi Subru and Arun.
> > > > >>
> > > > >> Thanks for driving 2.9 release. Great work!
> > > > >>
> > > > >> I installed cluster built from source.
> > > > >> - Ran few MR jobs with application priority enabled. Runs fine.
> > > > >> - Accessed new UI and it also seems fine.
> > > > >>
> > > > >> However I am also getting same issue as Rohith reported.
> > > > >> - Started an HA cluster
> > > > >> - Pushed RM to standby
> > > > >> - Pushed back RM to active then seeing an exception.
> > > > >>
> > > > >> org.apache.hadoop.ha.ServiceFailedException: RM could not
> > transition
> > > to
> > > > >> Active
> > > > >> at
> > > > >> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyE
> > > > >> lectorBasedElectorServic
> > > > >> e.becomeActive(ActiveStandbyElectorBasedElect
> > orService.java:146)
> > > > >> at
> > > > >> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(Activ
> > > > >> eStandbyElector.java:894
> > > > >> )
> > > > >>
> > > > >> Caused by: org.apache.zookeeper.KeeperException$NoAuthException:
> > > > >> KeeperErrorCode = NoAuth
> > > > >> at
> > > > >> org.apache.zookeeper.KeeperException.create(
> > KeeperException.java:113)
> > > > >> at org.apache.zookeeper.ZooKeeper.multiInternal(
> > > ZooKeeper.java:
> > > > >> 949)
> > > > >>
> > > > >> Will check and post more details,
> > > > >>
> > > > >> - Sunil
> > > > >>
> > > > >>
> > > > >> On Tue, Nov 7, 2017 at 12:47 PM Rohith Sharma K S <
> > > > >> rohithsharm...@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > Thanks Subru/Arun for the great work!
> > > > >> >
> > > > >> > Downloaded source and built from it. Deployed RM HA non-secured
> > > > cluster
> > > > >> > along with new YARN UI and ATSv2.
> > > > >> >
> > > > >> > I am facing basic RM HA switch issue after first time successful
> > > > start.
> > > > >> > *Can
> > > > >> > anyone else is facing this issue?*
> > > > >> >
> > > > >> > When RM is switched from ACTIVE to STANDBY to ACTIVE, RM never
> > > switch
> > > > to
> > > > >> > active successfully. Exception trace I see from the log is
> > > > >> >
> > > > >> > 2017-11-07 12:35:56,540 WARN org.apache.hadoop.ha.
> > > >

Re: Cutting branch-2.9

2017-11-03 Thread Subru Krishnan 
t-off branch-2.9.0 for 2.9.0 release
> >> work. In the mean time, branch-2.9 should be reserved for next 2.9 point
> >> release (2.9.1) and branch-2 should be reserved for next minor release
> >> (2.10.0 or whatever name it is). Thoughts?
> >>
> >> bq. @Junping, lets move the jdiff conversation to separate thread.
> >> Sure. I will reply jdiff in separated thread.
> >>
> >> Thanks,
> >>
> >> Junping
> >>
> >> 2017-10-31 13:44 GMT-07:00 Arun Suresh <asur...@apache.org>:
> >>
> >>> Hello folks
> >>>
> >>> We just cut branch-2.9 since all the critical/blocker issues are now
> >>> resolved.
> >>> We plan to perform some sanity checks for the rest of the week and cut
> >>> branch-2.9.0 and push out an RC0 by the end of the week.
> >>>
> >>> Kindly refrain from committing to branch-2.9 without giving us a heads
> >>> up.
> >>>
> >>> @Junping, lets move the jdiff conversation to separate thread.
> >>>
> >>> Cheers
> >>> -Arun/Subru
> >>>
> >>> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan <su...@apache.org>
> >>> wrote:
> >>>
> >>> > We want to give heads up that we are going to cut branch-2.9 tomorrow
> >>> > morning.
> >>> >
> >>> > We are down to 3 blockers and they all are close to being committed
> >>> > (thanks everyone):
> >>> > https://cwiki.apache.org/confluence/display/HADOOP/
> Hadoop+2.9+Release
> >>> >
> >>> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which
> >>> are
> >>> > close to completion.
> >>> >
> >>> > Folks who are working/reviewing these, kindly prioritize accordingly
> so
> >>> > that we can make the release on time.
> >>> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
> >>> >
> >>> > Thanks in advance!
> >>> >
> >>> > -Subru/Arun
> >>> >
> >>> >
> >>> >
> >>>
> >>
> >>
> >
>
>

Re: Cutting branch-2.9

2017-10-31 Thread Subru Krishnan 
Hi Junping,

We are planning to cut branch-2.9.0 on Thursday as there are couple of
non-blocking bug fixes coming in which we hope to get in tomorrow. Hope
that answers your question.

Thanks,
Subru

On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵 <dujunp...@gmail.com> wrote:

> Hi Arun/Subru,
> Thanks for updating on 2.9.0 release progress. A quick question here:
> are we planning to release from branch-2.9 directly?
> I doubt this as it seems against our current branch release practice (
> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of any
> confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release work.
> In the mean time, branch-2.9 should be reserved for next 2.9 point release
> (2.9.1) and branch-2 should be reserved for next minor release (2.10.0 or
> whatever name it is). Thoughts?
>
> bq. @Junping, lets move the jdiff conversation to separate thread.
> Sure. I will reply jdiff in separated thread.
>
> Thanks,
>
> Junping
>
> 2017-10-31 13:44 GMT-07:00 Arun Suresh <asur...@apache.org>:
>
>> Hello folks
>>
>> We just cut branch-2.9 since all the critical/blocker issues are now
>> resolved.
>> We plan to perform some sanity checks for the rest of the week and cut
>> branch-2.9.0 and push out an RC0 by the end of the week.
>>
>> Kindly refrain from committing to branch-2.9 without giving us a heads up.
>>
>> @Junping, lets move the jdiff conversation to separate thread.
>>
>> Cheers
>> -Arun/Subru
>>
>> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan <su...@apache.org>
>> wrote:
>>
>> > We want to give heads up that we are going to cut branch-2.9 tomorrow
>> > morning.
>> >
>> > We are down to 3 blockers and they all are close to being committed
>> > (thanks everyone):
>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>> >
>> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which
>> are
>> > close to completion.
>> >
>> > Folks who are working/reviewing these, kindly prioritize accordingly so
>> > that we can make the release on time.
>> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>> >
>> > Thanks in advance!
>> >
>> > -Subru/Arun
>> >
>> >
>> >
>>
>
>

Cutting branch-2.9

2017-10-30 Thread Subru Krishnan 
We want to give heads up that we are going to cut branch-2.9 tomorrow
morning.

We are down to 3 blockers and they all are close to being committed (thanks
everyone):
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release

There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are
close to completion.

Folks who are working/reviewing these, kindly prioritize accordingly so
that we can make the release on time.
https://issues.apache.org/jira/browse/YARN-7398?filter=12342468

Thanks in advance!

-Subru/Arun

2.9.0 status update (10/20/2017)

2017-10-20 Thread Subru Krishnan 
Today was the feature freeze date and we are glad to inform that all the
major planned features are merged in branch-2:
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Plannedfeatures
:

Kudos to everyone who pulled through multiple blockers and made this
happen. Special shoutout to Vrushali, Varun (both :)), Wangda, Inigo,
Sunil, and Jonathan.

I have set up a nightly build for branch-2 (hopefully):
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/

We have 5 blockers and 10 other jiras in development that have to be
addressed by *27th October 2017* following which we plan to cut branch-2.9:
Blockers - https://issues.apache.org/jira/issues/?filter=12342048
WIP JIRAs - https://issues.apache.org/jira/issues/?filter=12342468

We'll be following up with each of the above JIRAs individually next week,
lets make sure that we complete them by next Friday.

-Subru/Arun

Re: [VOTE] Merge Router-Based Federation (HDFS-10467) branch into trunk/branch-3

2017-10-02 Thread Subru Krishnan 
Thanks Inigo for driving this.

+1 (binding).

I have been involved during the design and have reviewed a couple of
patches. I feel this is a good addition and is well aligned with YARN
Federation (YARN-2915).

We deployed the HDFS Router in our test cluster and mapped it against 2
clusters - one running 3.0.0-beta1 and another 2.7.4 (to demonstrate
isolation) and it works fine.

-Subru

On Fri, Sep 29, 2017 at 11:58 AM, Iñigo Goiri  wrote:

> Hi all,
>
> Given that 3.0-beta1 is already cut, I’d like to call a vote for merging
> Router-Based Federation (HDFS-10467) to trunk and branch-3.
>
> The vote will run for 7 days as usual.
>
>
>
> We started the discussion about merging HDFS-10467 a few weeks ago [1] and
> got good feedback which we’ve incorporated already [2, 3, 4].
>
> There are a couple tasks left:
>
>- HDFS-12273 for the UI. This should be completed in the next couple
>days.
>- HDFS-12284 for adding security. We can move this for v2 if not
>completed.
>
> We have deployed this in production for 2.7 and we did a few tests with
> trunk a few months ago.
>
> This week, I’m rebasing to trunk (last one was a couple weeks ago) and
> test trunk in one of our test clusters.
>
>
> Finally, note that all the functionality is in the Router (a new
> component) so everything is isolated.
>
> In addition, no new APIs have been added and we rely fully in
> ClientProtocol.
>
>
>
> I’d like to thank the people at Microsoft (specially, Jason, Ricardo,
> Chris, Subru, Jakob, Carlo and Giovanni), Twitter (Ming and Gera), LinkedIn
> (Zhe, Erik and Konstantin), and Cloudera (Andrew and Manoj) for
> the discussion and the ideas.
>
> Special thanks to Chris Douglas for the thorough reviews!
>
>
>
> Regards,
> Inigo
>
>
>
> [1] http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201708.mbox/%
> 3CCAB1dGgogTu6kHtkkYeUycmNv-H3RupfPF4Cd7rpuFi6vHGdBLg%40mail.gmail.com%3E
>
> [2] https://issues.apache.org/jira/browse/HDFS-12381
>
> [3] https://issues.apache.org/jira/browse/HDFS-12430
>
> [4] https://issues.apache.org/jira/browse/HDFS-12450
>

2.9.0 Status Update [09/2017]

2017-09-29 Thread Subru Krishnan 
Folks,

Quick update on 2.9.0 release. Looks like we are looking at around a week
of delay in the feature freeze:
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap

For feature owners, where the status is:

   1. Green - no action required at this point.
   2. Orange - revert with confirmation on whether backporting from trunk
   to branch-2 is on track
   3. Red - can we get this in by week of Oct 9th?


There are about 20 blocker/critical issues open:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release

We have moved out all non-(blocker/critical) inactive (open) issues to the
next major release. Reach out to us if you feel any of those must be in
2.9.0.

For non-(blocker/critical) active (patch available) issues, can you update
the status. Consider moving out to next release if you anticipate delays.

Thanks,
Subru/Arun

[DISCUSS] Looking to a 2.9.0 release

2017-07-24 Thread Subru Krishnan 
Folks,

With the release for 2.8, we would like to look ahead to 2.9 release as
there are many features/improvements in branch-2 (about 1062 commits), that
are in need of a release vechile.

Here's our first cut of the proposal from the YARN side:

   1. Scheduler improvements (decoupling allocation from node heartbeat,
   allocation ID, concurrency fixes, LightResource etc).
   2. Timeline Service v2
   3. Opportunistic containers
   4. Federation

We would like to hear a formal list from HDFS & Hadoop (& MapReduce if any)
and will update the Roadmap wiki accordingly.

Considering our familiarity with the above mentioned YARN features, we
would like to volunteer as the co-RMs for 2.9.0.

We want to keep the timeline at 8-12 weeks to keep the release pragmatic.

Feedback?

-Subru/Arun