Re:
I have done the change. All committers, 2.8.2 release is supposed to be a stable/production release for branch-2.8. For commits to go for 2.8.2 release (only important and low risk bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For unimportant or high risk bug fixes/improvements, please commit to branch-2.8 (trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation! Thanks, Junping From: Junping Du Sent: Monday, July 24, 2017 10:36 AM To: Brahma Reddy Battula; Vinod Kumar Vavilapalli Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; humbed...@apache.org Subject: Re: Apache Hadoop 2.8.2 Release Plan Nice catch, Brahma. Actually, this is not supposed to be happen as we all should know the patch should firstly get landed on branch-2.8 (as well as trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check JIRAs claim to fixed in a specific release version with commits actually landing on the releasing branch before kicking off RC. So, I am not too worry about this mistaking behaviors as it happens in every releases. If no other concerns, I will do the branch update in next 30 minutes. Thanks, Junping From: Brahma Reddy BattulaSent: Sunday, July 23, 2017 1:50 AM To: Junping Du; Vinod Kumar Vavilapalli Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; humbed...@apache.org Subject: Re: Apache Hadoop 2.8.2 Release Plan Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two commits.Hope we'll not miss any commits which present in only in branch-2.8.2. From: Junping Du Sent: Saturday, July 22, 2017 5:57 AM To: Vinod Kumar Vavilapalli Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; humbed...@apache.org Subject: Re: Apache Hadoop 2.8.2 Release Plan Already get back from Daniel who is from ASF INFRA team, I plan to do following operations on next Monday morning: 1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8 2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 where we just released 2.8.1 from. I will also adjust fix version on all affected JIRA accordingly. If you have any concerns on above operations, please raise it before the end of this Sunday (7/23). Thanks, Junping From: Junping Du Sent: Friday, July 21, 2017 2:29 PM To: Vinod Kumar Vavilapalli Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe Subject: Re: Apache Hadoop 2.8.2 Release Plan Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669 Thanks, Junping From: Vinod Kumar Vavilapalli Sent: Friday, July 21, 2017 12:31 PM To: Junping Du Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe Subject: Re: Apache Hadoop 2.8.2 Release Plan Junping, If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how these things go. We can just continue to commit on branch-2.8 for now. I also think we should just follow up with ASF INFRA and clean up the branches - Delete branch-2.8.2 so that we can recreate it afresh a little later. - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private should be renamed to branch-2.8.1 Thanks +Vinod > On Jul 21, 2017, at 11:23 AM, Junping Du wrote: > > Thanks for suggestions, Jason and Kihwal! > +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 > cannot be abandoned/replaced (suspect all branches are read-only now), I will > manually merge all commits that not landed on 2.8.2 yet. > > Thanks, > > Junping > > From: Jason Lowe > Sent: Friday, July 21, 2017 8:17 AM > To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; > hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; > yarn-...@hadoop.apache.org > Subject: Re: Apache Hadoop 2.8.2 Release Plan > > +1 to base the 2.8.2 release off of the more recent activity on branch-2.8. > Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that > are in branch-2.8. There also are a lot of JIRAs that claim they are fixed > in 2.8.2 but are not in branch-2.8.2.
[DISCUSS] Looking to a 2.9.0 release
Folks, With the release for 2.8, we would like to look ahead to 2.9 release as there are many features/improvements in branch-2 (about 1062 commits), that are in need of a release vechile. Here's our first cut of the proposal from the YARN side: 1. Scheduler improvements (decoupling allocation from node heartbeat, allocation ID, concurrency fixes, LightResource etc). 2. Timeline Service v2 3. Opportunistic containers 4. Federation We would like to hear a formal list from HDFS & Hadoop (& MapReduce if any) and will update the Roadmap wiki accordingly. Considering our familiarity with the above mentioned YARN features, we would like to volunteer as the co-RMs for 2.9.0. We want to keep the timeline at 8-12 weeks to keep the release pragmatic. Feedback? -Subru/Arun
Re: Apache Hadoop 2.8.2 Release Plan
Nice catch, Brahma. Actually, this is not supposed to be happen as we all should know the patch should firstly get landed on branch-2.8 (as well as trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check JIRAs claim to fixed in a specific release version with commits actually landing on the releasing branch before kicking off RC. So, I am not too worry about this mistaking behaviors as it happens in every releases. If no other concerns, I will do the branch update in next 30 minutes. Thanks, Junping From: Brahma Reddy BattulaSent: Sunday, July 23, 2017 1:50 AM To: Junping Du; Vinod Kumar Vavilapalli Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; humbed...@apache.org Subject: Re: Apache Hadoop 2.8.2 Release Plan Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two commits.Hope we'll not miss any commits which present in only in branch-2.8.2. From: Junping Du Sent: Saturday, July 22, 2017 5:57 AM To: Vinod Kumar Vavilapalli Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; humbed...@apache.org Subject: Re: Apache Hadoop 2.8.2 Release Plan Already get back from Daniel who is from ASF INFRA team, I plan to do following operations on next Monday morning: 1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8 2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 where we just released 2.8.1 from. I will also adjust fix version on all affected JIRA accordingly. If you have any concerns on above operations, please raise it before the end of this Sunday (7/23). Thanks, Junping From: Junping Du Sent: Friday, July 21, 2017 2:29 PM To: Vinod Kumar Vavilapalli Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe Subject: Re: Apache Hadoop 2.8.2 Release Plan Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669 Thanks, Junping From: Vinod Kumar Vavilapalli Sent: Friday, July 21, 2017 12:31 PM To: Junping Du Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe Subject: Re: Apache Hadoop 2.8.2 Release Plan Junping, If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how these things go. We can just continue to commit on branch-2.8 for now. I also think we should just follow up with ASF INFRA and clean up the branches - Delete branch-2.8.2 so that we can recreate it afresh a little later. - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private should be renamed to branch-2.8.1 Thanks +Vinod > On Jul 21, 2017, at 11:23 AM, Junping Du wrote: > > Thanks for suggestions, Jason and Kihwal! > +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 > cannot be abandoned/replaced (suspect all branches are read-only now), I will > manually merge all commits that not landed on 2.8.2 yet. > > Thanks, > > Junping > > From: Jason Lowe > Sent: Friday, July 21, 2017 8:17 AM > To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; > hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; > yarn-...@hadoop.apache.org > Subject: Re: Apache Hadoop 2.8.2 Release Plan > > +1 to base the 2.8.2 release off of the more recent activity on branch-2.8. > Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that > are in branch-2.8. There also are a lot of JIRAs that claim they are fixed > in 2.8.2 but are not in branch-2.8.2. Having the 2.8.2 release be based on > recent activity in branch-2.8 would solve both of these issues, and we'd only > need to move the handful of JIRAs that have marked themselves correctly as > fixed in 2.8.3 to be fixed in 2.8.2. > > Jason > > >On Friday, July 21, 2017 10:01 AM, Kihwal Lee > wrote: > > > Thanks for driving the next 2.8 release, Junping. While I was committing a > blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but > missing in branch-2.8.2. Perhaps it is safer and easier to simply rebranch > 2.8.2. > Thanks,Kihwal > > On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du > wrote: > > Hi all, >Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get > released today which is
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/474/ [Jul 24, 2017 6:09:03 AM] (sunilg) YARN-6102. RMActiveService context to be updated with new RMContext on -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-hdfs-project/hadoop-hdfs-client Possible exposure of partially initialized object in org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At DFSClient.java:object in org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At DFSClient.java:[line 2888] org.apache.hadoop.hdfs.server.protocol.SlowDiskReports.equals(Object) makes inefficient use of keySet iterator instead of entrySet iterator At SlowDiskReports.java:keySet iterator instead of entrySet iterator At SlowDiskReports.java:[line 105] FindBugs : module:hadoop-hdfs-project/hadoop-hdfs Possible null pointer dereference in org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to return value of called method Dereferenced at JournalNode.java:org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to return value of called method Dereferenced at JournalNode.java:[line 302] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setClusterId(String) unconditionally sets the field clusterId At HdfsServerConstants.java:clusterId At HdfsServerConstants.java:[line 193] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForce(int) unconditionally sets the field force At HdfsServerConstants.java:force At HdfsServerConstants.java:[line 217] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForceFormat(boolean) unconditionally sets the field isForceFormat At HdfsServerConstants.java:isForceFormat At HdfsServerConstants.java:[line 229] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setInteractiveFormat(boolean) unconditionally sets the field isInteractiveFormat At HdfsServerConstants.java:isInteractiveFormat At HdfsServerConstants.java:[line 237] Possible null pointer dereference in org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, int, HardLink, boolean, File, List) due to return value of called method Dereferenced at DataStorage.java:org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, int, HardLink, boolean, File, List) due to return value of called method Dereferenced at DataStorage.java:[line 1339] Possible null pointer dereference in org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String, long) due to return value of called method Dereferenced at NNStorageRetentionManager.java:org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String, long) due to return value of called method Dereferenced at NNStorageRetentionManager.java:[line 258] Possible null pointer dereference in org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path, BasicFileAttributes) due to return value of called method Dereferenced at NNUpgradeUtil.java:org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path, BasicFileAttributes) due to return value of called method Dereferenced at NNUpgradeUtil.java:[line 133] Useless condition:argv.length >= 1 at this point At DFSAdmin.java:[line 2096] Useless condition:numBlocks == -1 at this point At ImageLoaderCurrent.java:[line 727] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Useless object stored in variable removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:[line 642] org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache() makes inefficient use of keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:[line 719] Hard coded reference to an absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:absolute pathname in