Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)

[Zhe Zhang]

+1 on RC1

- Verified signatures
- Built from source (RHEL7)
- Ran single node cluster and tested basic HDFS commands

Thanks for putting this together Jonathan!

On Mon, Oct 28, 2019 at 7:54 PM zhankun tang  wrote:

> Thanks for the efforts! Jonathan!
>
> +1 (non-binding) on RC1.
> - Set up a single node cluster with the binary tarball
> - Run Spark Pi and PySpark job
>
> BR,
> Zhankun
>
> On Mon, 28 Oct 2019 at 23:16, Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp>
> wrote:
>
> > Thanks for putting this up, Jonathan Hung.
> >
> > +1(non-binding)
> >
> > * verified signature and checksum.
> > * built source tarball by OpenJDK 7 on CentOS 7 with native profile
> > enabled.
> > * launched pseudo distributed cluster with security configurations.
> > * create encryption zone and put files in it.
> > * accessed files in encryption zone via httpfs.
> > * checked that file names containing '%' or '+' or ';' can be handled by
> > webhdfs.
> >
> > Masatake Iwasaki
> >
> > On 10/23/19 06:55, Jonathan Hung wrote:
> > > Hi folks,
> > >
> > > This is the second release candidate for the first release of Apache
> > Hadoop
> > > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It
> includes
> > > features such as:
> > >
> > > - User-defined resource types
> > > - Native GPU support as a schedulable resource type
> > > - Consistent reads from standby node
> > > - Namenode port based selective encryption
> > > - Improvements related to rolling upgrade support from 2.x to 3.x
> > > - Cost based fair call queue
> > >
> > > The RC1 artifacts are at:
> > http://home.apache.org/~jhung/hadoop-2.10.0-RC1/
> > >
> > > RC tag is release-2.10.0-RC1.
> > >
> > > The maven artifacts are hosted here:
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1243/
> > >
> > > My public key is available here:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm
> > PDT.
> > >
> > > Thanks,
> > > Jonathan Hung
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0)
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Zhe Zhang]

My late +1. Me and my colleagues have been involved in Submarine
development (in collaboration with our TonY project). I think Submarine
will be valuable as a new Apache project.

On Fri, Sep 6, 2019 at 10:57 PM Dinesh Chitlangia
 wrote:

> +1
>
> -Dinesh
>
>
>
>
> On Fri, Sep 6, 2019 at 11:23 PM 俊平堵  wrote:
>
> > +1. Please include me also.
> >
> > Thanks,
> >
> > Junping
> >
> > Wangda Tan  于2019年9月1日周日 下午1:19写道：
> >
> > > Hi all,
> > >
> > > As we discussed in the previous thread [1],
> > >
> > > I just moved the spin-off proposal to CWIKI and completed all TODO
> parts.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
> > >
> > > If you have interests to learn more about this. Please review the
> > proposal
> > > let me know if you have any questions/suggestions for the proposal.
> This
> > > will be sent to board post voting passed. (And please note that the
> > > previous voting thread [2] to move Submarine to a separate Github repo
> > is a
> > > necessary effort to move Submarine to a separate Apache project but not
> > > sufficient so I sent two separate voting thread.)
> > >
> > > Please let me know if I missed anyone in the proposal, and reply if
> you'd
> > > like to be included in the project.
> > >
> > > This voting runs for 7 days and will be concluded at Sep 7th, 11 PM
> PDT.
> > >
> > > Thanks,
> > > Wangda Tan
> > >
> > > [1]
> > >
> > >
> >
> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
> > > [2]
> > >
> > >
> >
> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
> > >
> >
>

Re: [VOTE] Merge YARN-8200 to branch-2 and branch-3.0

[Zhe Zhang]

+1 (binding)

As Jonathan said this feature in branch-2 has been running stably for over
a year. Therefore I’m supportive to the merge

On Mon, Aug 26, 2019 at 2:00 PM Jim Brennan 
wrote:

> +1 (non-binding).
> I have built branch-2 with the latest YARN-8200 patch
> (YARN-8200-branch-2.003.patch).  I ran all of the NM/RM tests and ran a few
> test jobs on a one-node cluster with default settings.
>
>
> On Mon, Aug 26, 2019 at 3:51 PM Oliver Hu  wrote:
>
> > +1 (non-binding)
> >
> > We have used this patch internally for more than a year to acquire GPU
> > reliably at LinkedIn. I don't think it is necessary to merge this to
> > branch-3.0 tho, as we are EOLing that branch.
> >
> > On Thu, Aug 22, 2019 at 4:43 PM Jonathan Hung 
> > wrote:
> >
> > > Hi folks,
> > >
> > > As per [1], starting a vote to merge YARN-8200 (and YARN-8200.branch3)
> > > feature branch to branch-2 (and branch-3.0).
> > >
> > > Vote runs for 7 days, to Thursday, Aug 29 5:00PM PDT.
> > >
> > > Thanks.
> > >
> > > [1]
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAHzWLgcX7f5Tr3q=csrqgysvpdf7mh-iu17femgx89dhr+1...@mail.gmail.com%3e
> > >
> > > Jonathan Hung
> > >
> >
>

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

[Zhe Zhang]

+1

On Mon, Feb 4, 2019 at 6:14 PM Jonathan Hung  wrote:

> Hello,
>
> Starting a vote based on the discuss thread [1] for moving branch-2
> precommit/nightly test builds to openjdk8. After this change, the test
> phase for precommit builds [2] and branch-2 nightly build [3] will run on
> openjdk8. To maintain source compatibility, these builds will still run
> their compile phase for branch-2 on openjdk7 as they do now (in addition to
> compiling on openjdk8).
>
> Vote will run for three business days until Thursday Feb 7 6:00PM PDT.
>
> [1]
>
> https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
>
> [2]
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
>
> [3]
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
>
> Jonathan Hung
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Propose to start new Hadoop sub project "submarine"

[Zhe Zhang]

+1, binding

On Mon, Feb 4, 2019 at 1:20 PM Takanobu Asanuma 
wrote:

> +1 (non-binding). Thanks, Wangda!
>
> - Takanobu
>
> on 2019/02/02 7:24, "Wangda Tan" wrote:
>
> Hi all,
>
> According to positive feedbacks from the thread [1]
>
> This is vote thread to start a new subproject named "hadoop-submarine"
> which follows the release process already established for ozone.
>
> The vote runs for usual 7 days, which ends at Feb 8th 5 PM PDT.
>
> Thanks,
> Wangda Tan
>
> [1]
>
> https://lists.apache.org/thread.html/f864461eb188bd12859d51b0098ec38942c4429aae7e4d001a633d96@%3Cyarn-dev.hadoop.apache.org%3E
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [DISCUSS] Making submarine to different release model like Ozone

[Zhe Zhang]

+1 on the proposal and looking forward to the progress of the project!

On Thu, Jan 31, 2019 at 10:51 PM Weiwei Yang  wrote:

> Thanks for proposing this Wangda, my +1 as well.
> It is amazing to see the progress made in Submarine last year, the
> community grows fast and quiet collaborative. I can see the reasons to get
> it release faster in its own cycle. And at the same time, the Ozone way
> works very well.
>
> —
> Weiwei
> On Feb 1, 2019, 10:49 AM +0800, Xun Liu , wrote:
> > +1
> >
> > Hello everyone,
> >
> > I am Xun Liu, the head of the machine learning team at Netease Research
> Institute. I quite agree with Wangda.
> >
> > Our team is very grateful for getting Submarine machine learning engine
> from the community.
> > We are heavy users of Submarine.
> > Because Submarine fits into the direction of our big data team's hadoop
> technology stack,
> > It avoids the needs to increase the manpower investment in learning
> other container scheduling systems.
> > The important thing is that we can use a common YARN cluster to run
> machine learning,
> > which makes the utilization of server resources more efficient, and
> reserves a lot of human and material resources in our previous years.
> >
> > Our team have finished the test and deployment of the Submarine and will
> provide the service to our e-commerce department (http://www.kaola.com/)
> shortly.
> >
> > We also plan to provides the Submarine engine in our existing YARN
> cluster in the next six months.
> > Because we have a lot of product departments need to use machine
> learning services,
> > for example:
> > 1) Game department (http://game.163.com/) needs AI battle training,
> > 2) News department (http://www.163.com) needs news recommendation,
> > 3) Mailbox department (http://www.163.com) requires anti-spam and
> illegal detection,
> > 4) Music department (https://music.163.com/) requires music
> recommendation,
> > 5) Education department (http://www.youdao.com) requires voice
> recognition,
> > 6) Massive Open Online Courses (https://open.163.com/) requires
> multilingual translation and so on.
> >
> > If Submarine can be released independently like Ozone, it will help us
> quickly get the latest features and improvements, and it will be great
> helpful to our team and users.
> >
> > Thanks hadoop Community!
> >
> >
> > > 在 2019年2月1日，上午2:53，Wangda Tan  写道：
> > >
> > > Hi devs,
> > >
> > > Since we started submarine-related effort last year, we received a lot
> of
> > > feedbacks, several companies (such as Netease, China Mobile, etc.) are
> > > trying to deploy Submarine to their Hadoop cluster along with big data
> > > workloads. Linkedin also has big interests to contribute a Submarine
> TonY (
> > > https://github.com/linkedin/TonY) runtime to allow users to use the
> same
> > > interface.
> > >
> > > From what I can see, there're several issues of putting Submarine under
> > > yarn-applications directory and have same release cycle with Hadoop:
> > >
> > > 1) We started 3.2.0 release at Sep 2018, but the release is done at Jan
> > > 2019. Because of non-predictable blockers and security issues, it got
> > > delayed a lot. We need to iterate submarine fast at this point.
> > >
> > > 2) We also see a lot of requirements to use Submarine on older Hadoop
> > > releases such as 2.x. Many companies may not upgrade Hadoop to 3.x in a
> > > short time, but the requirement to run deep learning is urgent to
> them. We
> > > should decouple Submarine from Hadoop version.
> > >
> > > And why we wanna to keep it within Hadoop? First, Submarine included
> some
> > > innovation parts such as enhancements of user experiences for YARN
> > > services/containerization support which we can add it back to Hadoop
> later
> > > to address common requirements. In addition to that, we have a big
> overlap
> > > in the community developing and using it.
> > >
> > > There're several proposals we have went through during Ozone merge to
> trunk
> > > discussion:
> > >
> https://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201803.mbox/%3ccahfhakh6_m3yldf5a2kq8+w-5fbvx5ahfgs-x1vajw8gmnz...@mail.gmail.com%3E
> > >
> > > I propose to adopt Ozone model: which is the same master branch,
> different
> > > release cycle, and different release branch. It is a great example to
> show
> > > agile release we can do (2 Ozone releases after Oct 2018) with less
> > > overhead to setup CI, projects, etc.
> > >
> > > *Links:*
> > > - JIRA: https://issues.apache.org/jira/browse/YARN-8135
> > > - Design doc
> > > <
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit
> >
> > > - User doc
> > > <
> https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/Index.html
> >
> > > (3.2.0
> > > release)
> > > - Blogposts, {Submarine} : Running deep learning workloads on Apache
> Hadoop
> > > <
> https://hortonworks.com/blog/submarine-running-deep-learning-workloads-apache-hadoop/
> >,
> > > (Chinese 

Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Zhe Zhang]

+1

Thanks for addressing concerns from the previous vote.

On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko 
wrote:

> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Zhe Zhang]

+1 (binding)

Thanks Konstantin for leading the merge effort!

I worked very closely with Chen, Konstantin, and Erik in the testing stage
and I feel confident that the feature has now completed designed
functionalities and has proven to be stable.

Great team work with contributors from multiple companies!

On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko 
wrote:

> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> There are a few outstanding issues:
> 1. Need to provide proper documentation - a user guide for the new feature
> 2. Need to fix automatic failover with ZKFC. Currently it does not doesn't
> know about ObserverNodes trying to convert them to SBNs.
> 3. Scale testing and performance fine-tuning
> 4. As testing progresses, we continue fixing non-critical bugs like
> HDFS-14116.
>
> I attached a unified patch to the umbrella jira for the review and Jenkins
> build.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 12.
>
> Thanks,
> --Konstantin
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

[Zhe Zhang]

+1 (binding)

- Downloaded source
- Verified checksum
- Built and started local cluster
- Checked YARN UI
- Performed basic HDFS operations

On Fri, Apr 13, 2018 at 1:40 PM Chen Liang <vagaryc...@gmail.com> wrote:

> Thanks for working on this Konstantin!
>
> +1 (non-binding)
>
> - verified checksum
> - built from source
> - started a single node HDFS cluster
> - performed basic operations of ls/mkdir/put/get
> - checked web UI
> - ran the MR job pi
>
> Regards,
> Chen
>
>
> > On Apr 9, 2018, at 4:14 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
> >
> > Hi everybody,
> >
> > This is the next dot release of Apache Hadoop 2.7 line. The previous one
> 2.7.5
> > was released on December 14, 2017.
> > Release 2.7.6 includes critical bug fixes and optimizations. See more
> > details in Release Note:
> > http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html
> >
> > The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/
> >
> > Please give it a try and vote on this thread. The vote will run for 5
> days
> > ending 04/16/2018.
> >
> > My up to date public key is available from:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Thanks,
> > --Konstantin
>
>
> ---------
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Zhe Zhang]

Great work Konstantin.

+1 (binding)
- Downloaded both source and binary tar files and verified checksums
- Deployed pseudo-distributed cluster and verified basic HDFS operations
(mkdir, copyFromLocal)
- Verified both NameNode and RM UIs

On Wed, Dec 13, 2017 at 12:05 PM Jonathan Hung <jyhung2...@gmail.com> wrote:

> Thanks Konstantin for working on this.
>
> +1 (non-binding)
> - Downloaded binary and verified md5
> - Deployed RM HA and tested failover
>
>
>
>
> Jonathan Hung
>
> On Wed, Dec 13, 2017 at 11:02 AM, Eric Payne <erichadoo...@yahoo.com
> .invalid
> > wrote:
>
> > Thanks for the hard work on this release, Konstantin.
> > +1 (binding)
> > - Built from source
> > - Verified that refreshing of queues works as expected.
> >
> > - Verified can run multiple users in a single queue
> > - Ran terasort test
> > - Verified that cross-queue preemption works as expected
> > Thanks. Eric Payne
> >
> >   From: Konstantin Shvachko <shv.had...@gmail.com>
> >  To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; "
> > hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; "
> > mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; "
> > yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>
> >  Sent: Thursday, December 7, 2017 9:22 PM
> >  Subject: [VOTE] Release Apache Hadoop 2.7.5 (RC1)
> >
> > Hi everybody,
> >
> > I updated CHANGES.txt and fixed documentation links.
> > Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
> >
> > This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
> > previous one 2.7.4 was release August 4, 2017.
> > Release 2.7.5 includes critical bug fixes and optimizations. See more
> > details in Release Note:
> > http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
> >
> > The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
> >
> > Please give it a try and vote on this thread. The vote will run for 5
> days
> > ending 12/13/2017.
> >
> > My up to date public key is available from:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Thanks,
> > --Konstantin
> >
> >
> >
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler configuration) to trunk, branch-3.0, branch-2

[Zhe Zhang]

+1 (binding)

LinkedIn has been running an internal version of OrgQueue for almost 2
years. It is an essential feature for running YARN scheduler on clusters
shared by multiple internal organizations.

I have participated in the YARN-5734 design and have regularly synced with
Jonathan on development and implementation details. I believe the feature
is of decent quality now and is ready to merge into trunk.

On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jyhung2...@gmail.com> wrote:

> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=pkec9dpggyx9rzf3n...@mail.gmail.com%3E
>
> Jonathan Hung
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

[jira] [Created] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility

[Zhe Zhang (JIRA)]

Zhe Zhang created MAPREDUCE-6937:


 Summary: Backport MAPREDUCE-6870 to branch-2 while preserving 
compatibility
 Key: MAPREDUCE-6937
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Zhe Zhang
Assignee: Erik Krogen






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [RESULT] [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Zhe Zhang]

Thanks Konstantin, great work!

On Sat, Aug 5, 2017 at 1:36 PM Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> My formal vote
> +1 (binding)
>
> I am glad to summarize that
> with 7 binding and 13 non-binding +1s and no -1s the vote for Apache
> Release 2.7.4 passes.
> Thank you everybody for contributing to the release, testing it, and
> voting.
>
> Binding +1s (7)
> Zhe Zhang
> Jason Lowe
> Eric Payne
> Sunil G
> Akira Ajisaka
> Chris Douglas
> Konstantin Shvachko
>
> Non-binding +1s (13)
> John Zhuge
> Surendra Lilhore
> Masatake Iwasaki
> Hanisha Koneru
> Chen Liang
> Fnu Ajay Kumar
> Brahma Reddy Battula
> Edwina Lu
> Ye Zhou
> Eric Badger
> Mingliang Liu
> Kuhu Shukla
> Erik Krogen
>
> Thanks,
> --Konstantin
>
> On Sat, Jul 29, 2017 at 4:29 PM, Konstantin Shvachko <shv.had...@gmail.com
> >
> wrote:
>
> > Hi everybody,
> >
> > Here is the next release of Apache Hadoop 2.7 line. The previous stable
> > release 2.7.3 was available since 25 August, 2016.
> > Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
> > critical bug fixes and major optimizations. See more details in Release
> > Note:
> > http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
> >
> > The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
> >
> > Please give it a try and vote on this thread. The vote will run for 5
> days
> > ending 08/04/2017.
> >
> > Please note that my up to date public key are available from:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > Please don't forget to refresh the page if you've been there recently.
> > There are other place on Apache sites, which may contain my outdated key.
> >
> > Thanks,
> > --Konstantin
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Zhe Zhang]

Quick question for Chris: was your +1 for the RC, or to support
Konstantin's statement regarding packaging?

Thanks,

On Mon, Jul 31, 2017 at 3:56 PM Chris Douglas <cdoug...@apache.org> wrote:

> On Mon, Jul 31, 2017 at 3:02 PM, Konstantin Shvachko
> <shv.had...@gmail.com> wrote:
> > For the packaging, here is the exact phrasing from the sited
> release-policy
> > document relevant to binaries:
> > "As a convenience to users that might not have the appropriate tools to
> > build a compiled version of the source, binary/bytecode packages MAY be
> > distributed alongside official Apache releases. In all such cases, the
> > binary/bytecode package MUST have the same version number as the source
> > release and MUST only add binary/bytecode files that are the result of
> > compiling that version of the source code release and its dependencies."
> > I don't think my binary package violates any of these.
>
> +1 The PMC VOTE applies to source code, only. If someone wants to
> rebuild the binary tarball with native libs and replace this one,
> that's fine.
>
> My reading of the above is that source code must be distributed with
> binaries, not that we omit the source code from binary releases... -C
>
> > But I'll upload an additional tar.gz with native bits and no src, as you
> > guys requested.
> > Will keep it as RC0 as there is no source code change and it comes from
> the
> > same build.
> > Hope this is satisfactory.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang <andrew.w...@cloudera.com>
> > wrote:
> >
> >> I agree with Brahma on the two issues flagged (having src in the binary
> >> tarball, missing native libs). These are regressions from prior
> releases.
> >>
> >> As an aside, "we release binaries as a convenience" doesn't relax the
> >> quality bar. The binaries are linked on our website and distributed
> through
> >> official Apache channels. They have to adhere to Apache release
> >> requirements. And, most users consume our work via Maven dependencies,
> >> which are binary artifacts.
> >>
> >> http://www.apache.org/legal/release-policy.html goes into this in more
> >> detail. A release must minimally include source packages, and can also
> >> include binary artifacts.
> >>
> >> Best,
> >> Andrew
> >>
> >> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko <
> >> shv.had...@gmail.com> wrote:
> >>
> >>> To avoid any confusion in this regard. I built RC0 manually in
> compliance
> >>> with Apache release policy
> >>> http://www.apache.org/legal/release-policy.html
> >>> I edited the HowToReleasePreDSBCR page to make sure people don't use
> >>> Jenkins option for building.
> >>>
> >>> A side note. This particular build is broken anyways, so no worries
> there.
> >>> I think though it would be useful to have it working for testing and
> as a
> >>> packaging standard.
> >>>
> >>> Thanks,
> >>> --Konstantin
> >>>
> >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
> >>> a...@effectivemachines.com
> >>> > wrote:
> >>>
> >>> >
> >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
> >>> shv.had...@gmail.com>
> >>> > wrote:
> >>> > >
> >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
> >>> >
> >>> > FYI:
> >>> >
> >>> > If you are using ASF Jenkins to create an ASF release
> >>> > artifact, it's pretty much an automatic vote failure as any such
> >>> release is
> >>> > in violation of ASF policy.
> >>> >
> >>> >
> >>>
> >>
> >>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Zhe Zhang]

gt; > > >>> I edited the HowToReleasePreDSBCR page to make sure people don't
> use
> > > >>> Jenkins option for building.
> > > >>>
> > > >>> A side note. This particular build is broken anyways, so no worries
> > > there.
> > > >>> I think though it would be useful to have it working for testing
> and
> > > as a
> > > >>> packaging standard.
> > > >>>
> > > >>> Thanks,
> > > >>> --Konstantin
> > > >>>
> > > >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
> > > >>> a...@effectivemachines.com
> > > >>> > wrote:
> > > >>>
> > > >>> >
> > > >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
> > > >>> shv.had...@gmail.com>
> > > >>> > wrote:
> > > >>> > >
> > > >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
> > > >>> >
> > > >>> > FYI:
> > > >>> >
> > > >>> > If you are using ASF Jenkins to create an ASF
> > release
> > > >>> > artifact, it's pretty much an automatic vote failure as any such
> > > >>> release is
> > > >>> > in violation of ASF policy.
> > > >>> >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > >
> >
>
>
>
> --
>
> *Zhou, Ye  **周晔*
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: About 2.7.4 Release

[Zhe Zhang]

Thanks for volunteering as RM Konstantin! The plan LGTM.

I've created a nightly Jenkins job for branch-2.7 (unit tests):
https://builds.apache.org/job/Hadoop-branch2.7-nightly/

On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hey guys,
>
> I and a few of my colleagues would like to help here and move 2.7.4 release
> forward. A few points in this regard.
>
> 1. Reading through this thread since March 1 I see that Vinod hinted on
> managing the release. Vinod, if you still want the job / have bandwidth
> will be happy to work with you.
> Otherwise I am glad to volunteer as the release manager.
>
> 2. In addition to current blockers and criticals, I would like to propose a
> few issues to be included in the release, see the list below. Those are
> mostly bug fixes and optimizations, which we already have in our internal
> branch and run in production. Plus one minor feature "node labeling", which
> we found very handy, when you have heterogeneous environments and mixed
> workloads, like MR and Spark.
>
> 3. For marking issues for the release I propose to
>  - set the target version to 2.7.4, and
>  - add a new label "release-blocker"
> That way we will know issues targeted for the release without reopening
> them for backports.
>
> 4. I see quite a few people are interested in the release. With all the
> help I think we can target to release by the end of May.
>
> Other things include fixing CHANGES.txt and fixing Jenkins build for 2.7.4
> branch.
>
> Thanks,
> --Konstantin
>
> ==  List of issue for 2.7.4  ===
> -- Backports
> HADOOP-12975 <https://issues.apache.org/jira/browse/HADOOP-12975>. Add du
> jitters
> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR batching
> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>. NPE when
> applying AvailableSpaceBlockPlacementPolicy
> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>. fsck removal
> of dot printing
> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
> space-balanced
> policy for balancer
> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>. abort
> balancer
> if upgrade in progress
> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>. skip small
> blocks in getBlocks
>
> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>. SLS simulator
> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>. SLS
> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>. SLS
> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>. SLS
>
> - Node labeling
> MAPREDUCE-6304 <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

[jira] [Created] (MAPREDUCE-6870) Add configuration for MR job to finish when all reducers are complete (even with unfinished mappers)

[Zhe Zhang (JIRA)]

Zhe Zhang created MAPREDUCE-6870:


 Summary: Add configuration for MR job to finish when all reducers 
are complete (even with unfinished mappers)
 Key: MAPREDUCE-6870
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6870
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 2.6.1
Reporter: Zhe Zhang


Even with MAPREDUCE-5817, there could still be cases where mappers get 
scheduled before all reducers are complete, but those mappers run for long 
time, even after all reducers are complete. This could hurt the performance of 
large MR jobs.

In some cases, mappers don't have any materialize-able outcome other than 
providing intermediate data to reducers. In that case, the job owner should 
have the config option to finish the job once all reducers are complete.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Updated 2.8.0-SNAPSHOT artifact

[Zhe Zhang]

I think recutting branch-2.8 from branch-2 is a good idea. We are running
2.6.x in production, and as a result we spend quite some effort for each
patch we want to backport (branch-2 => branch-2.8 => branch-2.7).

On Mon, Oct 31, 2016 at 1:20 AM Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> Thanks Akira and Karthik for replies.
>
>
> If there is no significant benefit over current branch-2.8 ,re-cutting
> will be an good idea, As we can reduce the maintenance effort.
>
>
>
> --Brahma Reddy Battula
>
>
> -Original Message-
> From: Akira Ajisaka [mailto:ajisa...@oss.nttdata.co.jp]
> Sent: 31 October 2016 14:06
> To: Karthik Kambatla
> Cc: Brahma Reddy Battula; Vinod Kumar Vavilapalli;
> common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: Updated 2.8.0-SNAPSHOT artifact
>
> Thanks Karthik for the comment.
>
> Pros:
>
> * (Probably) we can release 2.8.0 earlier.
>
>* New feature in 2.8.0 will be available sooner.
>* We can deprecate APIs sooner. After 2.8.0 is released, we can remove
> them from trunk.
>
> Cons:
>
> * Maintenance cost becomes higher because # of branches becomes more.
> * New feature in 2.9.0 will be available later
>
> I'm okay with either releasing current branch-2.8 or re-cutting it.
>
> Thoughts?
>
> Regards,
> Akira
>
> On 10/26/16 07:30, Karthik Kambatla wrote:
> > Is there value in releasing current branch-2.8? Aren't we better off
> > re-cutting the branch off of branch-2?
> >
> > On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka
> > <ajisa...@oss.nttdata.co.jp>
> > wrote:
> >
> >> It's almost a year since branch-2.8 has cut.
> >> I'm thinking we need to release 2.8.0 ASAP.
> >>
> >> According to the following list, there are 5 blocker and 6 critical
> issues.
> >> https://issues.apache.org/jira/issues/?filter=12334985
> >>
> >> Regards,
> >> Akira
> >>
> >>
> >> On 10/18/16 10:47, Brahma Reddy Battula wrote:
> >>
> >>> Hi Vinod,
> >>>
> >>> Any plan on first RC for branch-2.8 ? I think, it has been long time.
> >>>
> >>>
> >>>
> >>>
> >>> --Brahma Reddy Battula
> >>>
> >>> -Original Message-
> >>> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org]
> >>> Sent: 20 August 2016 00:56
> >>> To: Jonathan Eagles
> >>> Cc: common-...@hadoop.apache.org
> >>> Subject: Re: Updated 2.8.0-SNAPSHOT artifact
> >>>
> >>> Jon,
> >>>
> >>> That is around the time when I branched 2.8, so I guess you were
> >>> getting SNAPSHOT artifacts till then from the branch-2 nightly builds.
> >>>
> >>> If you need it, we can set up SNAPSHOT builds. Or just wait for the
> >>> first RC, which is around the corner.
> >>>
> >>> +Vinod
> >>>
> >>> On Jul 28, 2016, at 4:27 PM, Jonathan Eagles <jeag...@gmail.com>
> wrote:
> >>>>
> >>>> Latest snapshot is uploaded in Nov 2015, but checkins are still
> >>>> coming in quite frequently.
> >>>> https://repository.apache.org/content/repositories/snapshots/org/ap
> >>>> ach
> >>>> e/hadoop/hadoop-yarn-api/
> >>>>
> >>>> Are there any plans to start producing updated SNAPSHOT artifacts
> >>>> for current hadoop development lines?
> >>>>
> >>>
> >>>
> >>> 
> >>> - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>>
> >>>
> >>> 
> >>> - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>>
> >>>
> >>
> >> -
> >> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >>
> >>
> >
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Zhe Zhang]

+1 (non-binding)

Did the following on 7 RHEL 6.6 servers
- Downloaded and built from source
- Downloaded and verified checksum of the binary tar.gz file
- Setup a cluster with 1 NN and 6 DNs
- Tried regular HDFS commands
- Tried EC commands (listPolicies, getPolicy, setPolicy), they work fine
- Verified that with a 3-2 policy, 1.67x capacity is used. Below is the
output after copying the binary tar.gz file into an EC folder. The file is
318MB.

Configured Capacity: 3221225472 (3 GB)
Present Capacity: 3215348743 (2.99 GB)
DFS Remaining: 2655666176 (2.47 GB)
DFS Used: 559682567 (533.75 MB)

Thanks Allen for clarifying on the markdown files. I also verified the site
html files (content of the index.html, randomly selected some links).


On Tue, Aug 30, 2016 at 2:20 PM Eric Badger 
wrote:

> Well that's embarrassing. I had accidentally slightly renamed my
> log4j.properties file in my conf directory, so it was there, just not being
> read. Apologies for the unnecessary spam. With this and the public key from
> Andrew, I give my non-binding +1.
>
> Eric
>
>
>
> On Tuesday, August 30, 2016 4:11 PM, Allen Wittenauer <
> a...@effectivemachines.com> wrote:
>
>
> > On Aug 30, 2016, at 2:06 PM, Eric Badger 
> wrote:
> >
> >
> > WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be
> incomplete.
>
> ^^
>
>
> >
> > After running the above command, the RM UI showed a successful job, but
> as you can see, I did not have anything printed onto the command line.
> Hopefully this is just a misconfiguration on my part, but I figured that I
> would point it out just in case.
>
>
> It gave you a very important message in the output ...
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Zhe Zhang]

Thanks Andrew for the great work! It's really exciting to finally see a
Hadoop 3 RC.

I noticed CHANGES and RELEASENOTES markdown files which were not in
previous RCs like 2.7.3. What are good tools to verify them? I tried
reading them on IntelliJ but format looks odd.

I'm still testing the RC:
- Downloaded and verified checksum
- Built from source
- Will start small cluster and test simple programs, focusing on EC
functionalities

-- Zhe

On Tue, Aug 30, 2016 at 8:51 AM Andrew Wang 
wrote:

> Hi all,
>
> Thanks to the combined work of many, many contributors, here's an RC0 for
> 3.0.0-alpha1:
>
> http://home.apache.org/~wang/3.0.0-alpha1-RC0/
>
> alpha1 is the first in a series of planned alpha releases leading up to GA.
> The objective is to get an artifact out to downstreams for testing and to
> iterate quickly based on their feedback. So, please keep that in mind when
> voting; hopefully most issues can be addressed by future alphas rather than
> future RCs.
>
> Sorry for getting this out on a Tuesday, but I'd still like this vote to
> run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend
> if we lack the votes.
>
> Please try it out and let me know what you think.
>
> Best,
> Andrew
>

Re: [DISCUSS] Increased use of feature branches

[Zhe Zhang]

Thanks for the notes Andrew, Junping, Karthik.

Here are some of my understandings:

- Trunk is the "latest and greatest" of Hadoop. If a user starts using
Hadoop today, without legacy workloads, trunk is what he/she should use.
- Therefore, each commit to trunk should be transactional -- atomic,
consistent, isolated (from other uncommitted patches); I'm not so sure
about durability, Hadoop might be gone in 50 years :). As a committer, I
should be able to look at a patch and determine whether it's a
self-contained improvement of trunk, without looking at other uncommitted
patches.
- Some comments inline:

On Fri, Jun 10, 2016 at 6:56 AM Junping Du  wrote:

> Comparing with advantages, I believe the disadvantages of shipping any
> releases directly from trunk are more obvious and significant:
> - A lot of commits (incompatible, risky, uncompleted feature, etc.) have
> to wait to commit to trunk or put into a separated branch that could delay
> feature development progress as additional vote process get involved even
> the feature is simple and harmless.
>
Thanks Junping, those are valid concerns. I think we should clearly
separate incompatible with  uncompleted / half-done work in this
discussion. Whether people should commit incompatible changes to trunk is a
much more tricky question (related to trunk-incompat etc.). But per my
comment above, IMHO, *not committing uncompleted work to trunk* should be a
much easier principle to agree upon.


> - For small feature with only 1 or 2 commits, that need three +1 from PMCs
> will increase the bar largely for contributors who just start to contribute
> on Hadoop features but no such sufficient support.
>
Development overhead is another valid concern. I think our rule-of-thumb
should be that, small-medium new features should be proposed as a single
JIRA/patch (as we recently did for HADOOP-12666). If the complexity goes
beyond a single JIRA/patch, use a feature branch.


>
> Given these concerns, I am open to other options, like: proposed by Vinod
> or Chris, but rather than to release anything directly from trunk.
>
> - This point doesn't necessarily need to be resolved now though, since
> again we're still doing alphas.
> No. I think we have to settle down this first. Without a common agreed and
> transparent release process and branches in community, any release (alpha,
> beta) bits is only called a private release but not a official apache
> hadoop release (even alpha).
>
>
> Thanks,
>
> Junping
> 
> From: Karthik Kambatla 
> Sent: Friday, June 10, 2016 7:49 AM
> To: Andrew Wang
> Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: [DISCUSS] Increased use of feature branches
>
> Thanks for restarting this thread Andrew. I really hope we can get this
> across to a VOTE so it is clear.
>
> I see a few advantages shipping from trunk:
>
>- The lack of need for one additional backport each time.
>- Feature rot in trunk
>
> Instead of creating branch-3, I recommend creating branch-3.x so we can
> continue doing 3.x releases off branch-3 even after we move trunk to 4.x (I
> said it :))
>
> On Thu, Jun 9, 2016 at 11:12 PM, Andrew Wang 
> wrote:
>
> > Hi all,
> >
> > On a separate thread, a question was raised about 3.x branching and use
> of
> > feature branches going forward.
> >
> > We discussed this previously on the "Looking to a Hadoop 3 release"
> thread
> > that has spanned the years, with Vinod making this proposal (building on
> > ideas from others who also commented in the email thread):
> >
> >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201604.mbox/browser
> >
> > Pasting here for ease:
> >
> > On an unrelated note, offline I was pitching to a bunch of
> > contributors another idea to deal
> > with rotting trunk post 3.x: *Make 3.x releases off of trunk directly*.
> >
> > What this gains us is that
> >  - Trunk is always nearly stable or nearly ready for releases
> >  - We no longer have some code lying around in some branch (today’s
> > trunk) that is not releasable
> > because it gets mixed with other undesirable and incompatible changes.
> >  - This needs to be coupled with more discipline on individual
> > features - medium to to large
> > features are always worked upon in branches and get merged into trunk
> > (and a nearing release!)
> > when they are ready
> >  - All incompatible changes go into some sort of a trunk-incompat
> > branch and stay there till
> > we accumulate enough of those to warrant another major release.
> >
> > Regarding "trunk-incompat", since we're still in the alpha stage for
> 3.0.0,
> > there's no need for this branch yet. This aspect of Vinod's proposal was
> > still under a bit of discussion; Chris Douglas though we should cut a
> > branch-3 for the first 3.0.0 beta, which aligns with my original
> 

Re: CHANGES.txt is gone from trunk, branch-2, branch-2.8

[Zhe Zhang]

Great change! Saving ~1min for each commit. Thanks Andrew and Allen.

On Thu, Mar 3, 2016 at 9:32 PM Yongjun Zhang <yzh...@cloudera.com> wrote:

> That's nice, thanks Andrew and Allen.
>
> --Yongjun
>
> On Thu, Mar 3, 2016 at 9:11 PM, Andrew Wang <andrew.w...@cloudera.com>
> wrote:
>
> > Hi all,
> >
> > With the inclusion of HADOOP-12651 going back to branch-2.8, CHANGES.txt
> > and release notes are now generated by Yetus. I've gone ahead and deleted
> > the manually updated CHANGES.txt from trunk, branch-2, and branch-2.8
> > (HADOOP-11792). Many thanks to Allen for the releasedocmaker.py rewrite,
> > and the Yetus integration.
> >
> > I'll go ahead and update the HowToCommit and HowToRelease wiki pages, but
> > at a high-level, this means we no longer need to edit CHANGES.txt on new
> > commit, streamlining our commit process. CHANGES.txt updates will still
> be
> > necessary for backports to older release lines like 2.6.x and 2.7.x.
> >
> > Happy committing!
> >
> > Best,
> > Andrew
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap