[ANNOUNCE] Apache Hadoop 3.1.4 release
Greetings all, It gives me great pleasure to announce that the Apache Hadoop community has voted to release Apache Hadoop 3.1.4. Apache Hadoop 3.1.4 includes 308 fixes since the previous Hadoop 3.1.3 release. * For major changes included in Hadoop 3.1.4, please refer to Hadoop 3.1.4 main page [1]. * For more details about fixes in 3.1.4 release, please read the changelog [2] and release notes [3]. The release news is posted on the Hadoop website too, you can go to the downloads section directly [4]. Many thanks to everyone who contributed to the release, and everyone in the Apache Hadoop community! This release is a direct result of your great contributions. Many thanks to everyone who helped in this release process! [1] https://hadoop.apache.org/docs/r3.1.4/ [2] https://hadoop.apache.org/docs/r3.1.4/hadoop-project-dist/hadoop-common/release/3.1.4/CHANGES.3.1.4.html [3] https://hadoop.apache.org/docs/r3.1.4/hadoop-project-dist/hadoop-common/release/3.1.4/RELEASENOTES.3.1.4.html [4] https://hadoop.apache.org/releases.html Regards, Gabor Bota
Re: [VOTE] Release Apache Hadoop 3.1.4 (RC4)
Hi All, This is a recount of votes based on https://hadoop.apache.org/bylaws.html : Vote based on a Lazy Majority of active PMC members and Lazy Majority - A lazy majority vote requires 3 binding +1 votes and more binding +1 votes than -1 votes. So the following is the right number of votes: Binding +1s: - Masatake Iwasaki - Steve Loughran - Elek Marton Non-binding +1s: - Szilard Nemeth - Dinesh Chitlangia - Adam Antal - Mukund Madhav Thakur The RC is still voted +1 by the lazy majority, but I wanted to clarify this. Thank you Elek Marton for noticing this. Regards, Gabor On Mon, Aug 3, 2020 at 11:08 AM Gabor Bota wrote: > > Hi All, > > With 6 binding and 1 non-binding +1s and no -1s the vote for Apache > hadoop-3.1.4 Release passes. > > Thank you everybody for contributing to the release, testing, and voting. > > Binding +1s: > - Masatake Iwasaki > - Szilard Nemeth > - Steve Loughran > - Dinesh Chitlangia > - Adam Antal > - Elek, Marton > > Non-binding +1s: > - Mukund Madhav Thakur > > I'm going to work on staging the release. > > Thanks, > Gabor > > On Mon, Aug 3, 2020 at 10:03 AM Elek, Marton wrote: > > > > +1 (binding) > > > > * checked signature > > * built from source > > * deployed binary package to kubernetes > > * executed teragen with automatic tests [1] > > * checked "hadoop version" and compared with git revision > > * checked if the staging repository contains src packages > > > > > > Thanks the work (and the toughness) Gabor Bota. > > > > Marton > > > > > > [1]: https://github.com/elek/ozone-perf-env/tree/master/teragen-hdfs > > > > > > On 7/21/20 2:50 PM, Gabor Bota wrote: > > > Hi folks, > > > > > > I have put together a release candidate (RC4) for Hadoop 3.1.4. > > > > > > * > > > The RC includes in addition to the previous ones: > > > * fix for HDFS-15313. Ensure inodes in active filesystem are not > > > deleted during snapshot delete > > > * fix for YARN-10347. Fix double locking in > > > CapacityScheduler#reinitialize in branch-3.1 > > > (https://issues.apache.org/jira/browse/YARN-10347) > > > * the revert of HDFS-14941, as it caused > > > HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. > > > (https://issues.apache.org/jira/browse/HDFS-15421) > > > * HDFS-15323, as requested. > > > (https://issues.apache.org/jira/browse/HDFS-15323) > > > * > > > > > > The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC4/ > > > The RC tag in git is here: > > > https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC4 > > > The maven artifacts are staged at > > > https://repository.apache.org/content/repositories/orgapachehadoop-1275/ > > > > > > You can find my public key at: > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C > > > > > > Please try the release and vote. The vote will run for 8 weekdays, > > > until July 31. 2020. 23:00 CET. > > > > > > > > > Thanks, > > > Gabor > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.4 (RC4)
Hi All, With 6 binding and 1 non-binding +1s and no -1s the vote for Apache hadoop-3.1.4 Release passes. Thank you everybody for contributing to the release, testing, and voting. Binding +1s: - Masatake Iwasaki - Szilard Nemeth - Steve Loughran - Dinesh Chitlangia - Adam Antal - Elek, Marton Non-binding +1s: - Mukund Madhav Thakur I'm going to work on staging the release. Thanks, Gabor On Mon, Aug 3, 2020 at 10:03 AM Elek, Marton wrote: > > +1 (binding) > > * checked signature > * built from source > * deployed binary package to kubernetes > * executed teragen with automatic tests [1] > * checked "hadoop version" and compared with git revision > * checked if the staging repository contains src packages > > > Thanks the work (and the toughness) Gabor Bota. > > Marton > > > [1]: https://github.com/elek/ozone-perf-env/tree/master/teragen-hdfs > > > On 7/21/20 2:50 PM, Gabor Bota wrote: > > Hi folks, > > > > I have put together a release candidate (RC4) for Hadoop 3.1.4. > > > > * > > The RC includes in addition to the previous ones: > > * fix for HDFS-15313. Ensure inodes in active filesystem are not > > deleted during snapshot delete > > * fix for YARN-10347. Fix double locking in > > CapacityScheduler#reinitialize in branch-3.1 > > (https://issues.apache.org/jira/browse/YARN-10347) > > * the revert of HDFS-14941, as it caused > > HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. > > (https://issues.apache.org/jira/browse/HDFS-15421) > > * HDFS-15323, as requested. > > (https://issues.apache.org/jira/browse/HDFS-15323) > > * > > > > The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC4/ > > The RC tag in git is here: > > https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC4 > > The maven artifacts are staged at > > https://repository.apache.org/content/repositories/orgapachehadoop-1275/ > > > > You can find my public key at: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C > > > > Please try the release and vote. The vote will run for 8 weekdays, > > until July 31. 2020. 23:00 CET. > > > > > > Thanks, > > Gabor > > > > - > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop 3.1.4 (RC4)
Hi folks, I have put together a release candidate (RC4) for Hadoop 3.1.4. * The RC includes in addition to the previous ones: * fix for HDFS-15313. Ensure inodes in active filesystem are not deleted during snapshot delete * fix for YARN-10347. Fix double locking in CapacityScheduler#reinitialize in branch-3.1 (https://issues.apache.org/jira/browse/YARN-10347) * the revert of HDFS-14941, as it caused HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. (https://issues.apache.org/jira/browse/HDFS-15421) * HDFS-15323, as requested. (https://issues.apache.org/jira/browse/HDFS-15323) * The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC4/ The RC tag in git is here: https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC4 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1275/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C Please try the release and vote. The vote will run for 8 weekdays, until July 31. 2020. 23:00 CET. Thanks, Gabor - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.4 (RC3)
Thank you all for the suggestions and testing. As there's a data loss issue in the release, I've created a new RC with the patch included. I'll send the update soon. Regards, Gabor Bota On Thu, Jul 16, 2020 at 1:29 PM Stephen O'Donnell wrote: > > Hi Gabor, > > We recently discovered a HDFS data loss issue in any build which uses > snapshots containing HDFS-13101 but not including HDFS-15313. Unfortunately > 3.1.4 falls into this category: > > git log origin/branch-3.1.4 | egrep "HDFS-(15313|13101)" > HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. > Contributed by Shashikant Banerjee. > HDFS-13101. Yet another fsimage corruption related to snapshot. > Contributed by Shashikant Banerjee. > > See this comment for more information on the bug: > > https://issues.apache.org/jira/browse/HDFS-15313?focusedCommentId=17158140=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17158140 > > I think we should not make a release when we have a known data loss bug in > it. What do you think? > > I am going to commit HDFS-15313 onto branch-3.1 shortly, so maybe we should > cut a new RC after including that? > > Thanks, > > Stephen. > > On Mon, Jul 13, 2020 at 11:36 AM Gabor Bota > wrote: > > > Hi folks, > > > > I have put together a release candidate (RC3) for Hadoop 3.1.4. > > > > * > > The RC includes in addition to the previous ones: > > * fix of YARN-10347. Fix double locking in > > CapacityScheduler#reinitialize in branch-3.1 > > (https://issues.apache.org/jira/browse/YARN-10347) > > * the revert of HDFS-14941, as it caused > > HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. > > (https://issues.apache.org/jira/browse/HDFS-15421) > > * HDFS-15323, as requested. > > (https://issues.apache.org/jira/browse/HDFS-15323) > > * > > > > The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC3/ > > The RC tag in git is here: > > https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC3 > > The maven artifacts are staged at > > https://repository.apache.org/content/repositories/orgapachehadoop-1274/ > > > > You can find my public key at: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C > > > > Please try the release and vote. The vote will run for 7 weekdays, > > until July 22. 2020. 23:00 CET. > > > > > > Thanks, > > Gabor > > > > - > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop 3.1.4 (RC3)
Hi folks, I have put together a release candidate (RC3) for Hadoop 3.1.4. * The RC includes in addition to the previous ones: * fix of YARN-10347. Fix double locking in CapacityScheduler#reinitialize in branch-3.1 (https://issues.apache.org/jira/browse/YARN-10347) * the revert of HDFS-14941, as it caused HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. (https://issues.apache.org/jira/browse/HDFS-15421) * HDFS-15323, as requested. (https://issues.apache.org/jira/browse/HDFS-15323) * The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC3/ The RC tag in git is here: https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC3 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1274/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C Please try the release and vote. The vote will run for 7 weekdays, until July 22. 2020. 23:00 CET. Thanks, Gabor - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.4 (RC2)
Yes, sure. I'll do another RC for next week. Thank you all for working on this! On Thu, Jul 9, 2020 at 8:20 AM Masatake Iwasaki wrote: > > Hi Gabor Bota, > > I committed the fix of YARN-10347 to branch-3.1. > I think this should be blocker for 3.1.4. > Could you cherry-pick it to branch-3.1.4 and cut a new RC? > > Thanks, > Masatake Iwasaki > > On 2020/07/08 23:31, Masatake Iwasaki wrote: > > Thanks Steve and Prabhu for the information. > > > > The cause turned out to be locking in CapacityScheduler#reinitialize. > > I think the method is called after transitioning to active stat if > > RM-HA is enabled. > > > > I filed YARN-10347 and created PR. > > > > > > Masatake Iwasaki > > > > > > On 2020/07/08 16:33, Prabhu Joseph wrote: > >> Hi Masatake, > >> > >> The thread is waiting for a ReadLock, we need to check what the > >> other > >> thread holding WriteLock is blocked on. > >> Can you get three consecutive complete jstack of ResourceManager > >> during the > >> issue. > >> > >>>> I got no issue if RM-HA is disabled. > >> Looks RM is not able to access Zookeeper State Store. Can you check if > >> there is any connectivity issue between RM and Zookeeper. > >> > >> Thanks, > >> Prabhu Joseph > >> > >> > >> On Mon, Jul 6, 2020 at 2:44 AM Masatake Iwasaki > >> > >> wrote: > >> > >>> Thanks for putting this up, Gabor Bota. > >>> > >>> I'm testing the RC2 on 3 node docker cluster with NN-HA and RM-HA > >>> enabled. > >>> ResourceManager reproducibly blocks on submitApplication while > >>> launching > >>> example MR jobs. > >>> Does anyone run into the same issue? > >>> > >>> The same configuration worked for 3.1.3. > >>> I got no issue if RM-HA is disabled. > >>> > >>> > >>> "IPC Server handler 1 on default port 8032" #167 daemon prio=5 > >>> os_prio=0 > >>> tid=0x7fe91821ec50 nid=0x3b9 waiting on condition > >>> [0x7fe901bac000] > >>> java.lang.Thread.State: WAITING (parking) > >>> at sun.misc.Unsafe.park(Native Method) > >>> - parking to wait for <0x85d37a40> (a > >>> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > >>> at > >>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > >>> at > >>> > >>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > >>> > >>> at > >>> > >>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) > >>> > >>> at > >>> > >>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) > >>> > >>> at > >>> > >>> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) > >>> > >>> at > >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkAndGetApplicationPriority(CapacityScheduler.java:2521) > >>> > >>> at > >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:417) > >>> > >>> at > >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:342) > >>> > >>> at > >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:678) > >>> > >>> at > >>> > >>> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:277) > >>> > >>> at > >>> > >>> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:563) > >>> > >>> at > >>> > >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) >
[VOTE] Release Apache Hadoop 3.1.4 (RC2)
Hi folks, I have put together a release candidate (RC2) for Hadoop 3.1.4. The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC2/ The RC tag in git is here: https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC2 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1269/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C Please try the release and vote. The vote will run for 5 weekdays, until July 6. 2020. 23:00 CET. The release includes the revert of HDFS-14941, as it caused HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. (https://issues.apache.org/jira/browse/HDFS-15421) The release includes HDFS-15323, as requested. (https://issues.apache.org/jira/browse/HDFS-15323) Thanks, Gabor - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.4 (RC1)
Correction: I meant "withdraw this vote now for RC1" and "create an RC2". Ayush, I'll add it if it applies without conflict. Regards, Gabor On Wed, Jun 24, 2020 at 1:01 PM Ayush Saxena wrote: > > Hi Gabor, > As you are going ahead with another RC, > Please include : HDFS-15323 as well if possible. > > https://issues.apache.org/jira/browse/HDFS-15323 > > Remember tagging you there but at that time RC0 was up, try if that could > make into the release as well. > > Thanx!!! > -Ayush > > On Wed, 24 Jun 2020 at 16:16, Gabor Bota > wrote: > > > Thanks for looking into this Akira, Kihwal! > > > > > > I noted that it is a hard to create situation described in HDFS-14941. > > The issue created by HDFS-14941 would be even harder to fix in > > HDFS-15421, test it, prove that it's stable, etc.. > > That's why I will do a revert of HDFS-14941 and create an RC3. > > > > > > > > * I withdraw this vote now for RC2 because of that blocker issue > > (HDFS-15421). I will create an RC3 with HDFS-14941 reverted. * > > > > Regards, > > Gabor > > > > On Tue, Jun 23, 2020 at 4:59 PM Kihwal Lee > > wrote: > > > > > > Gabor, > > > If you want to release asap, you can simply revert HDFS-14941 in the > > > release branch for now. It is causing the issue and was committed after > > > 3.1.3. This causes failure of the automated upgrade process and namenode > > > memory leak. > > > > > > Kihwal > > > > > > On Tue, Jun 23, 2020 at 8:47 AM Akira Ajisaka > > wrote: > > > > > > > Hi Gabor, > > > > > > > > Thank you for your work! > > > > > > > > Kihwal reported IBR leak in standby NameNode: > > > > https://issues.apache.org/jira/browse/HDFS-15421. > > > > I think this is a blocker and this affects 3.1.4-RC1. Would you check > > this? > > > > > > > > Best regards, > > > > Akira > > > > > > > > On Mon, Jun 22, 2020 at 10:26 PM Gabor Bota > > > .invalid> > > > > wrote: > > > > > > > > > Hi folks, > > > > > > > > > > I have put together a release candidate (RC1) for Hadoop 3.1.4. > > > > > > > > > > The RC is available at: > > > > http://people.apache.org/~gabota/hadoop-3.1.4-RC1/ > > > > > The RC tag in git is here: > > > > > https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC1 > > > > > The maven artifacts are staged at > > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1267/ > > > > > > > > > > You can find my public key at: > > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > and > > http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C > > > > > > > > > > Please try the release and vote. The vote will run for 5 weekdays, > > > > > until June 30. 2020. 23:00 CET. > > > > > > > > > > Thanks, > > > > > Gabor > > > > > > > > > > - > > > > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > > > > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > > > > > > > > > > > > > > > > > > - > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.4 (RC1)
Thanks for looking into this Akira, Kihwal! I noted that it is a hard to create situation described in HDFS-14941. The issue created by HDFS-14941 would be even harder to fix in HDFS-15421, test it, prove that it's stable, etc.. That's why I will do a revert of HDFS-14941 and create an RC3. * I withdraw this vote now for RC2 because of that blocker issue (HDFS-15421). I will create an RC3 with HDFS-14941 reverted. * Regards, Gabor On Tue, Jun 23, 2020 at 4:59 PM Kihwal Lee wrote: > > Gabor, > If you want to release asap, you can simply revert HDFS-14941 in the > release branch for now. It is causing the issue and was committed after > 3.1.3. This causes failure of the automated upgrade process and namenode > memory leak. > > Kihwal > > On Tue, Jun 23, 2020 at 8:47 AM Akira Ajisaka wrote: > > > Hi Gabor, > > > > Thank you for your work! > > > > Kihwal reported IBR leak in standby NameNode: > > https://issues.apache.org/jira/browse/HDFS-15421. > > I think this is a blocker and this affects 3.1.4-RC1. Would you check this? > > > > Best regards, > > Akira > > > > On Mon, Jun 22, 2020 at 10:26 PM Gabor Bota > .invalid> > > wrote: > > > > > Hi folks, > > > > > > I have put together a release candidate (RC1) for Hadoop 3.1.4. > > > > > > The RC is available at: > > http://people.apache.org/~gabota/hadoop-3.1.4-RC1/ > > > The RC tag in git is here: > > > https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC1 > > > The maven artifacts are staged at > > > https://repository.apache.org/content/repositories/orgapachehadoop-1267/ > > > > > > You can find my public key at: > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C > > > > > > Please try the release and vote. The vote will run for 5 weekdays, > > > until June 30. 2020. 23:00 CET. > > > > > > Thanks, > > > Gabor > > > > > > - > > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > > > > > > > > - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop 3.1.4 (RC1)
Hi folks, I have put together a release candidate (RC1) for Hadoop 3.1.4. The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC1/ The RC tag in git is here: https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC1 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1267/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C Please try the release and vote. The vote will run for 5 weekdays, until June 30. 2020. 23:00 CET. Thanks, Gabor - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.4 (RC0)
Hi All, Voting has ended with: 0 binding +1s 0 non-binding +1s (including mine) 0 binding -1s 0 non-binding -1s I'm going to proceed and ask for you to vote another time on the RC1 soon. Regards, Gabor On Wed, May 13, 2020 at 8:17 AM Mukund Madhav Thakur wrote: > Yes , I have azure-auth-keys only. And abfs tests work fine in the current > trunk branch with my same azure-auth-keys.xml file. > Fyi: I am running mvn -T 1C -Dparallel-tests=abfs clean verify > *One thing I noticed here is , in this branch there are no ABFS related > classes.* > > @Gabor Bota I did the packaging of the release > as well using > > mvn package -Pdist -DskipTests -Dmaven.javadoc.skip=true -DskipShade > and ran some hadoop fs commands as well. All good there. > > On Wed, May 13, 2020 at 9:10 AM Masatake Iwasaki < > iwasak...@oss.nttdata.co.jp> wrote: > >> > Also I am trying to run the abfs tests but all tests are getting skipped >> > even if I think I have right auth-keys.xml. I will debug this and >> update. >> >> hadoop-azure expects azure-auth-keys.xml instead of auth-keys.xml? >> I think it should be consistent with other FS modules.. >> >> Masatake Iwasaki >> >> On 2020/05/13 0:33, Mukund Madhav Thakur wrote: >> > I compiled and ran s3 tests using >> > >> > mvn clean verify -Ds3guard -Ddynamo -Dauth. I see some failures. I ran >> > these separately as well but it still fails for me. >> > >> > >> > Also I am trying to run the abfs tests but all tests are getting skipped >> > even if I think I have right auth-keys.xml. I will debug this and >> update. >> > >> > >> > >> > [*ERROR*] *Failures: * >> > >> > [*ERROR*] * >> > >> ITestS3AMiscOperations.testEmptyFileChecksums:147->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 >> > checksums expected: but >> > was:* >> > >> > [*ERROR*] * >> > >> ITestS3AMiscOperations.testNonEmptyFileChecksumsUnencrypted:199->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 >> > checksums expected: but >> > was:* >> > >> > [*INFO*] >> > >> > [*ERROR*] *Tests run: 12, Failures: 2, Errors: 0, Skipped: 0* >> > >> > >> > [*ERROR*] *Errors: * >> > >> > [*ERROR*] * >> > >> ITestS3GuardToolDynamoDB.testDynamoDBInitDestroyCycle:224->AbstractS3GuardToolTestBase.exec:286->AbstractS3GuardToolTestBase.exec:308 >> > » AWSServiceIO* >> > >> > >> > >> > >> > On Tue, May 5, 2020 at 11:48 PM Steve Loughran >> >> > wrote: >> > >> >> mvn -T 1 -Phadoop-3.2 -Dhadoop.version=3.1.4 -Psnapshots-and-staging >> >> -Phadoop-cloud,yarn,kinesis-asl,yarn clean install -DskipTests >> >> >> >> Then a test run of the cloud bits >> >> >> >> mvn -T 1 -Phadoop-3.2 -Dhadoop.version=3.1.4 -Psnapshots-and-staging >> >> -Phadoop-cloud,yarn,kinesis-asl test --pl hadoop-cloud >> >> >> >> And I got a guava binding stack trace (joy!) >> >> >> >> >> >> CommitterBindingSuite: >> >> *** RUN ABORTED *** >> >>java.lang.NoSuchMethodError: >> >> >> >> >> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V >> >>at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) >> >>at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) >> >>at >> >> org.apache.spark.internal.io >> >> .cloud.CommitterBindingSuite.newJob(CommitterBindingSuite.scala:89) >> >>at >> >> org.apache.spark.internal.io >> >> >> .cloud.CommitterBindingSuite.$anonfun$new$1(CommitterBindingSuite.scala:55) >> >>at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) >> >>at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) >> >>at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) >> >>at org.scalatest.Transformer.apply(Transformer.scala:22) >> >>at org.scalatest.Transformer.apply(Transformer.scala:20) >> >>at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186) >> >>... >> >> >> >> Fix: spark's guava version needs to be bumped up. There's a bigger >> patch >> >> for that, but I've got a minor PR which lets someone change it on the >> maven >> >> CLI to
[VOTE] Release Apache Hadoop 3.1.4 (RC0)
Hi folks, I have put together a release candidate (RC0) for Hadoop 3.1.4. The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC0/ The RC tag in git is here: https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC0 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1266/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C Please try the release and vote. The vote will run for 5 weekdays, until May 11. 2020. 23:00 CET. Thanks, Gabor - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Hadoop 3.1.4 Release Code Freeze
Hi All, The code freeze for Hadoop 3.1.4 release is planned for today. There some issues still open for that version. These are the open issues with 3.1.4 target/fix version: * https://issues.apache.org/jira/browse/YARN-8234 * https://issues.apache.org/jira/browse/HDFS-15205 * https://issues.apache.org/jira/browse/HADOOP-16341 * https://issues.apache.org/jira/browse/HDFS-15272 * https://issues.apache.org/jira/browse/HADOOP-16193 * https://issues.apache.org/jira/browse/YARN-8257 * https://issues.apache.org/jira/browse/YARN-8417 * https://issues.apache.org/jira/browse/HADOOP-16917 Please check if your issue is a blocker. I will move these issues to 3.1.5 target/fix version after 04.23. 10AM CET. I also commented on each issue separately. Release info: * https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.1+Release * https://cwiki.apache.org/confluence/display/HADOOP/Roadmap Regards, Gabor - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Hadoop 3.1.4 Release Plan Proposal
Hi All, Based on the discussion on the topic "Hadoop 2019 Release Planning" I volunteer to do the next 3.1 Hadoop release, version 3.1.4. You can find the blocker/critical issues and all issues targeted for 3.1.4 under the following cwiki page: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.1+Release Right now there are 217 total fixed issues[1] and 6 blocker/critical targeted[2] for the release. If you have any other commits you want to backport to 3.1 branch until the feature or code freeze, please add 3.1.4 as the target release in jira. My proposed timeline for the release is the following: * Feature Freeze Date: Wednesday, 11 March 2020 * Code Freeze Date: Wednesday, 22 April 2020 * Release Date: Wednesday, 29 April 2020 Please let me know if you have any suggestions. Regards, Gabor Bota References: [1] project in (HADOOP, MAPREDUCE, HDFS, YARN) AND priority in (Blocker, Critical) AND "Target Version/s" = 3.1.4 ORDER BY priority DESC, updated DESC [2] project in (HADOOP, MAPREDUCE, HDFS, YARN) AND fixVersion = 3.1.4 ORDER BY priority DESC, updated DESC
Re: [DISCUSS] Hadoop 2019 Release Planning
I'm volunteering to be the release manager for 3.1.4. Where can I find a wiki page for the 3.1 line? I can not find one here: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines On Tue, Jan 7, 2020 at 3:41 PM Steve Loughran wrote: > I'm thinking of doing a backport of most of the hadoop-aws changes to > branch-3.2, for the next 3.2.x release; they are all self contained and > will benefit many (they will need to cope with the older mockito version, > but I have to deal with that in-house already). > > one change is the new openFile() builder API. I'd like to wrap that up with > a little improvement https://issues.apache.org/jira/browse/HADOOP-16759; > That way for all releases with the API, it's consistent. > > (that withStatus() feature gives extra performance and ensures that > etag/version can be used to get the explicit version you want.) > > On Tue, Jan 7, 2020 at 2:18 AM Akira Ajisaka wrote: > > > > I am interested on 3.3 release ..will act as RM .will update the wiki > as > > well.. > > > > Thanks Brahma for your reply. I'll help you as co-RM. > > We will send announcements (cutting branches, code freeze, and so on) in > > another thread. > > > > Thanks, > > Akira > > > > On Tue, Jan 7, 2020 at 4:32 AM Wangda Tan wrote: > > > > > Hi guys, > > > > > > Thanks for the update and for volunteering to be RM. > > > > > > I just did a quick check: > > > 3.1.4 has 52 patches resolved. (3.1.3 Released on Oct 21) > > > 3.2.2 has 46 patches resolved. (3.2.1 Released on Sep 22) > > > 3.3.0 has .. many patches sitting here so we definitely need a release. > > > > > > If Akira and Brahma you guys can be co-RMs for 3.3.0 that would be > great. > > > > > > Hadoop 3.2.1 is released on Sep 22 which is 3+ months ago, and I saw > > > community started to have large prod deployment on 3.2.x, Gabor if you > > have > > > bandwidth to help releases, I think we can do 3.2.2 first then 3.1.4. > > > > > > Thoughts? > > > - Wangda > > > > > > On Mon, Jan 6, 2020 at 5:50 AM Brahma Reddy Battula > > > > wrote: > > > > > >> Thanks Akira for resuming this.. > > >> > > >> I am interested on 3.3 release ..will act as RM .will update the wiki > > as > > >> well.. > > >> > > >> > > >> > > >> On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota > .invalid> > > >> wrote: > > >> > > >>> I'm interested in doing a release of hadoop. > > >>> The version we need an RM is 3.1.3 right? What's the target date for > > >>> that? > > >>> > > >>> Thanks, > > >>> Gabor > > >>> > > >>> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka > > >>> wrote: > > >>> > > >>> > Thank you Wangda. > > >>> > > > >>> > Now it's 2020. Let's release Hadoop 3.3.0. > > >>> > I created a wiki page for tracking blocker/critical issues for > 3.3.0 > > >>> and > > >>> > I'll check the issues in the list. > > >>> > > > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release > > >>> > If you find blocker/critical issues in trunk, please set the target > > >>> version > > >>> > to 3.3.0 for tracking. > > >>> > > > >>> > > We still need RM for 3.3.0 and 3.1.3. > > >>> > I can work as a release manager for 3.3.0. Is there anyone who > wants > > >>> to be > > >>> > a RM? > > >>> > > > >>> > Thanks and regards, > > >>> > Akira > > >>> > > > >>> > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang < > tangzhan...@gmail.com> > > >>> > wrote: > > >>> > > > >>> > > Thanks Wangda for bring this up! > > >>> > > > > >>> > > I ran the submarine 0.2.0 release before with a lot of help from > > >>> folks > > >>> > > especially Sunil. :D > > >>> > > And this time I would like to help to release the 3.1.4. Thanks! > > >>> > > > > >>> > > BR, > > >>> > > Zhankun > > >>> > > > > >>> > > Hui Fei 于2019年8月16日 周五下午7:19写道： > > >>> >
Re: [DISCUSS] Hadoop 2019 Release Planning
I'm interested in doing a release of hadoop. The version we need an RM is 3.1.3 right? What's the target date for that? Thanks, Gabor On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka wrote: > Thank you Wangda. > > Now it's 2020. Let's release Hadoop 3.3.0. > I created a wiki page for tracking blocker/critical issues for 3.3.0 and > I'll check the issues in the list. > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release > If you find blocker/critical issues in trunk, please set the target version > to 3.3.0 for tracking. > > > We still need RM for 3.3.0 and 3.1.3. > I can work as a release manager for 3.3.0. Is there anyone who wants to be > a RM? > > Thanks and regards, > Akira > > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang > wrote: > > > Thanks Wangda for bring this up! > > > > I ran the submarine 0.2.0 release before with a lot of help from folks > > especially Sunil. :D > > And this time I would like to help to release the 3.1.4. Thanks! > > > > BR, > > Zhankun > > > > Hui Fei 于2019年8月16日 周五下午7:19写道： > > > > > Hi Wangda, > > > Thanks for bringing this up! > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade is a > > > problem. > > > Hope commiters watch and review these issues, Thanks > > > https://issues.apache.org/jira/browse/HDFS-13596 > > > https://issues.apache.org/jira/browse/HDFS-14396 > > > > > > Wangda Tan 于2019年8月10日周六 上午10:59写道： > > > > > > > Hi all, > > > > > > > > Hope this email finds you well > > > > > > > > I want to hear your thoughts about what should be the release plan > for > > > > 2019. > > > > > > > > In 2018, we released: > > > > - 1 maintenance release of 2.6 > > > > - 3 maintenance releases of 2.7 > > > > - 3 maintenance releases of 2.8 > > > > - 3 releases of 2.9 > > > > - 4 releases of 3.0 > > > > - 2 releases of 3.1 > > > > > > > > Total 16 releases in 2018. > > > > > > > > In 2019, by far we only have two releases: > > > > - 1 maintenance release of 3.1 > > > > - 1 minor release of 3.2. > > > > > > > > However, the community put a lot of efforts to stabilize features of > > > > various release branches. > > > > There're: > > > > - 217 fixed patches in 3.1.3 [1] > > > > - 388 fixed patches in 3.2.1 [2] > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!) > > > > > > > > I think it is the time to do maintenance releases of 3.1/3.2 and do a > > > minor > > > > release for 3.3.0. > > > > > > > > In addition, I saw community discussion to do a 2.8.6 release for > > > security > > > > fixes. > > > > > > > > Any other releases? I think there're release plans for Ozone as well. > > And > > > > please add your thoughts. > > > > > > > > Volunteers welcome! If you have interests to run a release as Release > > > > Manager (or co-Resource Manager), please respond to this email thread > > so > > > we > > > > can coordinate. > > > > > > > > Thanks, > > > > Wangda Tan > > > > > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed > > AND > > > > fixVersion = 3.1.3 > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed > > AND > > > > fixVersion = 3.2.1 > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed > > AND > > > > fixVersion = 3.3.0 > > > > > > > > > >
Re: [DISCUSS] Prefer Github PR Integration over patch in JIRA
Although we will use github with PRs, I'd still prefer adding a +1 as a jira comment stating which PR was the last and approved one among the many. On Tue, Jul 23, 2019 at 11:22 AM Steve Loughran wrote: > On Mon, Jul 22, 2019 at 7:29 PM Eric Badger > wrote: > > > Where would JIRA fit into the PR workflow? Would we file JIRAs just to > > track github PRs and have all of the discussion on the PR? > > > > > Every code contribution needs its JIRA for: tracking, release notes, cross > referencing; every committed patch needs that JIRA reference. > > Reviews of specific patches go into the PRs > > I actually think discussion about overall direction of work is better in > the JIRA, because a complex piece of work can have multiple PRs: different > attempts where when you need to rebase its best to create a new one so the > old discussion is still linked to specific lines of code, and when > different people take a PR and contribute their own work. > > That split of comments across >1 PR is one of the costs of using github for > review. >
Re: [VOTE] Force "squash and merge" option for PR merge on github UI
+1 Good idea.
On Wed, Jul 17, 2019 at 9:37 AM Ayush Saxena wrote:
> Thanks Marton, Makes Sense +1
>
> > On 17-Jul-2019, at 11:37 AM, Elek, Marton wrote:
> >
> > Hi,
> >
> > Github UI (ui!) helps to merge Pull Requests to the proposed branch.
> > There are three different ways to do it [1]:
> >
> > 1. Keep all the different commits from the PR branch and create one
> > additional merge commit ("Create a merge commit")
> >
> > 2. Squash all the commits and commit the change as one patch ("Squash
> > and merge")
> >
> > 3. Keep all the different commits from the PR branch but rebase, merge
> > commit will be missing ("Rebase and merge")
> >
> >
> >
> > As only the option 2 is compatible with the existing development
> > practices of Hadoop (1 issue = 1 patch = 1 commit), I call for a lazy
> > consensus vote: If no objections withing 3 days, I will ask INFRA to
> > disable the options 1 and 3 to make the process less error prone.
> >
> > Please let me know, what do you think,
> >
> > Thanks a lot
> > Marton
> >
> > ps: Personally I prefer to merge from local as it enables to sign the
> > commits and do a final build before push. But this is a different story,
> > this proposal is only about removing the options which are obviously
> > risky...
> >
> > ps2: You can always do any kind of merge / commits from CLI, for example
> > to merge a feature branch together with keeping the history.
> >
> > [1]:
> >
> https://help.github.com/en/articles/merging-a-pull-request#merging-a-pull-request-on-github
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
