[jira] [Created] (HADOOP-14676) Wrong default value for "fs.du.interval"
Konstantin Shvachko created HADOOP-14676: Summary: Wrong default value for "fs.du.interval" Key: HADOOP-14676 URL: https://issues.apache.org/jira/browse/HADOOP-14676 Project: Hadoop Common Issue Type: Bug Components: common, conf, fs Affects Versions: 2.6.1 Reporter: Konstantin Shvachko According to {{core-default.xml}} the default value of {{fs.du.interval = 60 sec}}. But the implementation of {{DF}} substitutes 3 sec instead. The problem is that {{DF}} uses outdated constant {{DF.DF_INTERVAL_DEFAULT}} instead of the correct one {{CommonConfigurationKeysPublic.FS_DU_INTERVAL_DEFAULT}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Pre-Commit build is failing
What stuff? Is there a jira? It did work like a week ago. Is it a new Yetus requirement. Anyways I can commit a change to fix the build on our side. Just need to know what is missing. Thanks, --Konst On Fri, Jul 21, 2017 at 5:50 PM, Allen Wittenauerwrote: > > > On Jul 21, 2017, at 5:46 PM, Konstantin Shvachko > wrote: > > > > + d...@yetus.apache.org > > > > Guys, could you please take a look. Seems like Yetus problem with > > pre-commit build for branch-2.7. > > > branch-2.7 is missing stuff in .gitignore.
Re: Pre-Commit build is failing
+ d...@yetus.apache.org Guys, could you please take a look. Seems like Yetus problem with pre-commit build for branch-2.7. Thanks, --Konstantin On Thu, Jul 20, 2017 at 7:19 PM, Brahma Reddy Battula < brahmareddy.batt...@huawei.com> wrote: > Looks this problem is in only branc-2.7.. > > > --Brahma Reddy Battula > > From: Brahma Reddy Battula > Sent: 21 July 2017 09:36 > To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org > Subject: Pre-Commit build is failing > Importance: High > > Looks pre-commit build is failing with following error. > > > /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS- > Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh: > line 87: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS- > Build/patchprocess/patch-dryrun.log: No such file or directory > /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS- > Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh: > line 98: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS- > Build/patchprocess/patch-dryrun.log: No such file or directory > /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS- > Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh: > line 87: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS- > Build/patchprocess/patch-dryrun.log: No such file or directory > /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS- > Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh: > line 98: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS- > Build/patchprocess/patch-dryrun.log: No such file or directory > > > > Reference : > > https://builds.apache.org/view/PreCommit%20Builds/job/ > PreCommit-HDFS-Build/20362/console > > > > > --Brahma Reddy Battula > >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/471/ [Jul 20, 2017 8:15:46 AM] (haibochen) YARN-6685. Add job count in to SLS JSON input format. (Yufei Gu via [Jul 20, 2017 9:03:04 AM] (jlowe) YARN-6837. Null LocalResource visibility or resource type can crash the [Jul 20, 2017 2:12:21 PM] (kihwal) HADOOP-14659. UGI getShortUserName does not need to search the Subject. [Jul 20, 2017 2:47:25 PM] (shv) HDFS-11472. Fix inconsistent replica size after a data pipeline failure. [Jul 20, 2017 6:23:13 PM] (weichiu) HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by [Jul 20, 2017 8:45:09 PM] (jzhuge) HDFS-12166. Do not deprecate HTTPFS_TEMP. Contributed by John Zhuge. -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-hdfs-project/hadoop-hdfs-client Possible exposure of partially initialized object in org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At DFSClient.java:object in org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At DFSClient.java:[line 2888] org.apache.hadoop.hdfs.server.protocol.SlowDiskReports.equals(Object) makes inefficient use of keySet iterator instead of entrySet iterator At SlowDiskReports.java:keySet iterator instead of entrySet iterator At SlowDiskReports.java:[line 105] FindBugs : module:hadoop-hdfs-project/hadoop-hdfs Possible null pointer dereference in org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to return value of called method Dereferenced at JournalNode.java:org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to return value of called method Dereferenced at JournalNode.java:[line 302] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setClusterId(String) unconditionally sets the field clusterId At HdfsServerConstants.java:clusterId At HdfsServerConstants.java:[line 193] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForce(int) unconditionally sets the field force At HdfsServerConstants.java:force At HdfsServerConstants.java:[line 217] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForceFormat(boolean) unconditionally sets the field isForceFormat At HdfsServerConstants.java:isForceFormat At HdfsServerConstants.java:[line 229] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setInteractiveFormat(boolean) unconditionally sets the field isInteractiveFormat At HdfsServerConstants.java:isInteractiveFormat At HdfsServerConstants.java:[line 237] Possible null pointer dereference in org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, int, HardLink, boolean, File, List) due to return value of called method Dereferenced at DataStorage.java:org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, int, HardLink, boolean, File, List) due to return value of called method Dereferenced at DataStorage.java:[line 1339] Possible null pointer dereference in org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String, long) due to return value of called method Dereferenced at NNStorageRetentionManager.java:org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String, long) due to return value of called method Dereferenced at NNStorageRetentionManager.java:[line 258] Possible null pointer dereference in org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path, BasicFileAttributes) due to return value of called method Dereferenced at NNUpgradeUtil.java:org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path, BasicFileAttributes) due to return value of called method Dereferenced at NNUpgradeUtil.java:[line 133] Useless condition:argv.length >= 1 at this point At DFSAdmin.java:[line 2096] Useless condition:numBlocks == -1 at this point At ImageLoaderCurrent.java:[line 727] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Useless object stored in variable removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:[line 642]
Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)
Hi Sanjay, Actually I was not aware of that work… This seems to be a better way of achieving some of the same things we do externally to the DN process. I will look into reimplementing some parts on top of this; seems it should just require some very small extensions to DataNodeCluster. Thank you very much for the pointer! Erik On 7/21/17, 11:01 AM, "sanjay Radia"wrote: Erik Great stuff. BTW did you build on top of the “simulated data nodes” in HDFS which has a way to storing only the length of data (but not real data)? That work allowed supplementing with a matching editsLog for the NN. Your approach of using a real image has the advantage of being able to replay traces from audit logs. (Ref https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java) thanks sanjay > On Jul 20, 2017, at 10:42 AM, Erik Krogen wrote: > > forking off of the 2.7.4 release thread to answer this question about > Dynamometer > > Dynamometer is a tool developed at LinkedIn for scale testing HDFS, > specifically the NameNode. We have been using it for some time now and have > recently been making some enhancements to ease of use and reproducibility. > We hope to post a blog post sometime in the not-too-distant future, and > also to open source it. I can provide some details here given that we have > been leveraging it as part of our 2.7.4 release / upgrade process (in > addition to previous upgrades). > > The basic idea is to get full-scale black-box testing of the HDFS NN while > using significantly less (~10%) hardware than a real cluster of that size > would require. We use real NN images from our at-scale clusters paired with > some logic to fake out DNs into thinking they are storing data when they > are not, allowing us to stuff more DNs onto each machine. Since we use a > real image, we can replay real traces (collected from audit logs) to > compare actual production performance vs. performance on this simulated > cluster (with additional tuning, different version, etc.). We leverage YARN > to manage setting up this cluster and to replay the traces. > > Happy to answer questions. > > Erik > > On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko > wrote: > >> Hi Tianyi, >> >> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working >> on this project right now, I'll let him elaborate. >> Erik, you should probably respond on Apache dev list, as I think it could >> be interesting for other people as well, asince we planned to open source >> it. You can fork the "About 2.7.4 Release" thread with a new subject and >> give some details about Dynamometer there. >> >> Thanks, >> --Konstantin >> >> On Wed, Jul 19, 2017 at 1:40 AM, 何天一 wrote: >> >>> Hi, Shavachko. >>> >>> You mentioned an internal tool called Dynamometer to test NameNode >>> performance earlier in the 2.7.4 release thread. >>> I wonder if you could share some ideas behind the tool. Or is there a >>> plan to bring Dynamometer to open source community? >>> >>> Thanks. >>> >>> BR, >>> Tianyi >>> >>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko >>> wrote: >>> Hi everybody. We have been doing some internal testing of Hadoop 2.7.4. The testing is going well. Did not find any major issues on our workloads. Used an internal tool called Dynamometer to check NameNode performance on real cluster traces. Good. Overall test cluster performance looks good. Some more testing is still going on. I plan to build an RC next week. If there are no objection. Thanks, --Konst On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko < shv.had...@gmail.com> wrote: > Hey guys. > > An update on 2.7.4 progress. > We are down to 4 blockers. There is some work remaining on those. > https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814 > Would be good if people could follow up on review comments. > > I looked through nightly Jenkins build results for 2.7.4 both on Apache > Jenkins and internal. > Some test fail intermittently, but there no consistent failures. I filed > HDFS-11985 to track some of them. > https://issues.apache.org/jira/browse/HDFS-11985 > I do not currently consider these failures as blockers. LMK if some of > them are. > > We started
Re: Apache Hadoop 2.8.2 Release Plan
Already get back from Daniel who is from ASF INFRA team, I plan to do following operations on next Monday morning: 1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8 2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 where we just released 2.8.1 from. I will also adjust fix version on all affected JIRA accordingly. If you have any concerns on above operations, please raise it before the end of this Sunday (7/23). Thanks, Junping From: Junping DuSent: Friday, July 21, 2017 2:29 PM To: Vinod Kumar Vavilapalli Cc: Kihwal Lee; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe Subject: Re: Apache Hadoop 2.8.2 Release Plan Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669 Thanks, Junping From: Vinod Kumar Vavilapalli Sent: Friday, July 21, 2017 12:31 PM To: Junping Du Cc: Kihwal Lee; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe Subject: Re: Apache Hadoop 2.8.2 Release Plan Junping, If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how these things go. We can just continue to commit on branch-2.8 for now. I also think we should just follow up with ASF INFRA and clean up the branches - Delete branch-2.8.2 so that we can recreate it afresh a little later. - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private should be renamed to branch-2.8.1 Thanks +Vinod > On Jul 21, 2017, at 11:23 AM, Junping Du wrote: > > Thanks for suggestions, Jason and Kihwal! > +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 > cannot be abandoned/replaced (suspect all branches are read-only now), I will > manually merge all commits that not landed on 2.8.2 yet. > > Thanks, > > Junping > > From: Jason Lowe > Sent: Friday, July 21, 2017 8:17 AM > To: Kihwal Lee; Junping Du; common-dev@hadoop.apache.org; > hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; > yarn-...@hadoop.apache.org > Subject: Re: Apache Hadoop 2.8.2 Release Plan > > +1 to base the 2.8.2 release off of the more recent activity on branch-2.8. > Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that > are in branch-2.8. There also are a lot of JIRAs that claim they are fixed > in 2.8.2 but are not in branch-2.8.2. Having the 2.8.2 release be based on > recent activity in branch-2.8 would solve both of these issues, and we'd only > need to move the handful of JIRAs that have marked themselves correctly as > fixed in 2.8.3 to be fixed in 2.8.2. > > Jason > > >On Friday, July 21, 2017 10:01 AM, Kihwal Lee > wrote: > > > Thanks for driving the next 2.8 release, Junping. While I was committing a > blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but > missing in branch-2.8.2. Perhaps it is safer and easier to simply rebranch > 2.8.2. > Thanks,Kihwal > > On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du > wrote: > > Hi all, >Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get > released today which is a special security release. Now, we should work > towards 2.8.2 release which aim for production deployment. The focus > obviously is to fix blocker/critical issues [2], bug-fixes and *no* features > / improvements. We currently have 13 blocker/critical issues, and 10 of them > are Patch Available. > > I plan to cut an RC in a month - target for releasing before end of Aug., to > give enough time for outstanding blocker / critical issues. Will start moving > out any tickets that are not blockers and/or won't fit the timeline. For > progress of releasing effort, please refer our release wiki [2]. > > Please share thoughts if you have any. Thanks! > > Thanks, > > Junping > > [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x > [2] 2.8 Release wiki: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release > > > From: Vinod Kumar Vavilapalli > Sent: Thursday, July 20, 2017 1:05 PM > To: gene...@hadoop.apache.org > Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released > > Hi all, > > The Apache Hadoop PMC has released version 2.8.1. You can get it from this > page: http://hadoop.apache.org/releases.html#Download > This is a security release in the 2.8.0 release line. It consists of 2.8.0 > plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1. > > Please note that 2.8.x release line continues to be not yet ready for > production use. Critical issues are being ironed out via
Re: Apache Hadoop 2.8.2 Release Plan
Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669 Thanks, Junping From: Vinod Kumar VavilapalliSent: Friday, July 21, 2017 12:31 PM To: Junping Du Cc: Kihwal Lee; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe Subject: Re: Apache Hadoop 2.8.2 Release Plan Junping, If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how these things go. We can just continue to commit on branch-2.8 for now. I also think we should just follow up with ASF INFRA and clean up the branches - Delete branch-2.8.2 so that we can recreate it afresh a little later. - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private should be renamed to branch-2.8.1 Thanks +Vinod > On Jul 21, 2017, at 11:23 AM, Junping Du wrote: > > Thanks for suggestions, Jason and Kihwal! > +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 > cannot be abandoned/replaced (suspect all branches are read-only now), I will > manually merge all commits that not landed on 2.8.2 yet. > > Thanks, > > Junping > > From: Jason Lowe > Sent: Friday, July 21, 2017 8:17 AM > To: Kihwal Lee; Junping Du; common-dev@hadoop.apache.org; > hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; > yarn-...@hadoop.apache.org > Subject: Re: Apache Hadoop 2.8.2 Release Plan > > +1 to base the 2.8.2 release off of the more recent activity on branch-2.8. > Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that > are in branch-2.8. There also are a lot of JIRAs that claim they are fixed > in 2.8.2 but are not in branch-2.8.2. Having the 2.8.2 release be based on > recent activity in branch-2.8 would solve both of these issues, and we'd only > need to move the handful of JIRAs that have marked themselves correctly as > fixed in 2.8.3 to be fixed in 2.8.2. > > Jason > > >On Friday, July 21, 2017 10:01 AM, Kihwal Lee > wrote: > > > Thanks for driving the next 2.8 release, Junping. While I was committing a > blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but > missing in branch-2.8.2. Perhaps it is safer and easier to simply rebranch > 2.8.2. > Thanks,Kihwal > > On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du > wrote: > > Hi all, >Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get > released today which is a special security release. Now, we should work > towards 2.8.2 release which aim for production deployment. The focus > obviously is to fix blocker/critical issues [2], bug-fixes and *no* features > / improvements. We currently have 13 blocker/critical issues, and 10 of them > are Patch Available. > > I plan to cut an RC in a month - target for releasing before end of Aug., to > give enough time for outstanding blocker / critical issues. Will start moving > out any tickets that are not blockers and/or won't fit the timeline. For > progress of releasing effort, please refer our release wiki [2]. > > Please share thoughts if you have any. Thanks! > > Thanks, > > Junping > > [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x > [2] 2.8 Release wiki: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release > > > From: Vinod Kumar Vavilapalli > Sent: Thursday, July 20, 2017 1:05 PM > To: gene...@hadoop.apache.org > Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released > > Hi all, > > The Apache Hadoop PMC has released version 2.8.1. You can get it from this > page: http://hadoop.apache.org/releases.html#Download > This is a security release in the 2.8.0 release line. It consists of 2.8.0 > plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1. > > Please note that 2.8.x release line continues to be not yet ready for > production use. Critical issues are being ironed out via testing and > downstream adoption. Production users should wait for a subsequent release in > the 2.8.x line. > > Thanks > +Vinod > > > - > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Apache Hadoop 2.8.2 Release Plan
Junping, If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how these things go. We can just continue to commit on branch-2.8 for now. I also think we should just follow up with ASF INFRA and clean up the branches - Delete branch-2.8.2 so that we can recreate it afresh a little later. - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private should be renamed to branch-2.8.1 Thanks +Vinod > On Jul 21, 2017, at 11:23 AM, Junping Duwrote: > > Thanks for suggestions, Jason and Kihwal! > +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 > cannot be abandoned/replaced (suspect all branches are read-only now), I will > manually merge all commits that not landed on 2.8.2 yet. > > Thanks, > > Junping > > From: Jason Lowe > Sent: Friday, July 21, 2017 8:17 AM > To: Kihwal Lee; Junping Du; common-dev@hadoop.apache.org; > hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; > yarn-...@hadoop.apache.org > Subject: Re: Apache Hadoop 2.8.2 Release Plan > > +1 to base the 2.8.2 release off of the more recent activity on branch-2.8. > Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that > are in branch-2.8. There also are a lot of JIRAs that claim they are fixed > in 2.8.2 but are not in branch-2.8.2. Having the 2.8.2 release be based on > recent activity in branch-2.8 would solve both of these issues, and we'd only > need to move the handful of JIRAs that have marked themselves correctly as > fixed in 2.8.3 to be fixed in 2.8.2. > > Jason > > >On Friday, July 21, 2017 10:01 AM, Kihwal Lee > wrote: > > > Thanks for driving the next 2.8 release, Junping. While I was committing a > blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but > missing in branch-2.8.2. Perhaps it is safer and easier to simply rebranch > 2.8.2. > Thanks,Kihwal > > On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du > wrote: > > Hi all, >Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get > released today which is a special security release. Now, we should work > towards 2.8.2 release which aim for production deployment. The focus > obviously is to fix blocker/critical issues [2], bug-fixes and *no* features > / improvements. We currently have 13 blocker/critical issues, and 10 of them > are Patch Available. > > I plan to cut an RC in a month - target for releasing before end of Aug., to > give enough time for outstanding blocker / critical issues. Will start moving > out any tickets that are not blockers and/or won't fit the timeline. For > progress of releasing effort, please refer our release wiki [2]. > > Please share thoughts if you have any. Thanks! > > Thanks, > > Junping > > [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x > [2] 2.8 Release wiki: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release > > > From: Vinod Kumar Vavilapalli > Sent: Thursday, July 20, 2017 1:05 PM > To: gene...@hadoop.apache.org > Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released > > Hi all, > > The Apache Hadoop PMC has released version 2.8.1. You can get it from this > page: http://hadoop.apache.org/releases.html#Download > This is a security release in the 2.8.0 release line. It consists of 2.8.0 > plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1. > > Please note that 2.8.x release line continues to be not yet ready for > production use. Critical issues are being ironed out via testing and > downstream adoption. Production users should wait for a subsequent release in > the 2.8.x line. > > Thanks > +Vinod > > > - > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Apache Hadoop 2.8.2 Release Plan
Thanks for suggestions, Jason and Kihwal! +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 cannot be abandoned/replaced (suspect all branches are read-only now), I will manually merge all commits that not landed on 2.8.2 yet. Thanks, Junping From: Jason LoweSent: Friday, July 21, 2017 8:17 AM To: Kihwal Lee; Junping Du; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: Re: Apache Hadoop 2.8.2 Release Plan +1 to base the 2.8.2 release off of the more recent activity on branch-2.8. Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that are in branch-2.8. There also are a lot of JIRAs that claim they are fixed in 2.8.2 but are not in branch-2.8.2. Having the 2.8.2 release be based on recent activity in branch-2.8 would solve both of these issues, and we'd only need to move the handful of JIRAs that have marked themselves correctly as fixed in 2.8.3 to be fixed in 2.8.2. Jason On Friday, July 21, 2017 10:01 AM, Kihwal Lee wrote: Thanks for driving the next 2.8 release, Junping. While I was committing a blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but missing in branch-2.8.2. Perhaps it is safer and easier to simply rebranch 2.8.2. Thanks,Kihwal On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du wrote: Hi all, Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get released today which is a special security release. Now, we should work towards 2.8.2 release which aim for production deployment. The focus obviously is to fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We currently have 13 blocker/critical issues, and 10 of them are Patch Available. I plan to cut an RC in a month - target for releasing before end of Aug., to give enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won't fit the timeline. For progress of releasing effort, please refer our release wiki [2]. Please share thoughts if you have any. Thanks! Thanks, Junping [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x [2] 2.8 Release wiki: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release From: Vinod Kumar Vavilapalli Sent: Thursday, July 20, 2017 1:05 PM To: gene...@hadoop.apache.org Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released Hi all, The Apache Hadoop PMC has released version 2.8.1. You can get it from this page: http://hadoop.apache.org/releases.html#Download This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1. Please note that 2.8.x release line continues to be not yet ready for production use. Critical issues are being ironed out via testing and downstream adoption. Production users should wait for a subsequent release in the 2.8.x line. Thanks +Vinod - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14675) Path object disallows access to S3 objects with // in their names
Andi created HADOOP-14675: - Summary: Path object disallows access to S3 objects with // in their names Key: HADOOP-14675 URL: https://issues.apache.org/jira/browse/HADOOP-14675 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.7.1 Environment: Plain vanila Hadoop 2.7. Reporter: Andi Seems that there's explicit code to disallow FS paths with //. In our environment this means that direct access using url's like s3a://bucket/data//part1 are inaccessible. Why does this code exist? Can it be removed? https://github.com/apache/hadoop/blob/6fefb8f4a486b1d8071a7071e9e96f0c6299752d/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java#L289-L289 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)
Erik Great stuff. BTW did you build on top of the “simulated data nodes” in HDFS which has a way to storing only the length of data (but not real data)? That work allowed supplementing with a matching editsLog for the NN. Your approach of using a real image has the advantage of being able to replay traces from audit logs. (Ref https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java) thanks sanjay > On Jul 20, 2017, at 10:42 AM, Erik Krogen> wrote: > > forking off of the 2.7.4 release thread to answer this question about > Dynamometer > > Dynamometer is a tool developed at LinkedIn for scale testing HDFS, > specifically the NameNode. We have been using it for some time now and have > recently been making some enhancements to ease of use and reproducibility. > We hope to post a blog post sometime in the not-too-distant future, and > also to open source it. I can provide some details here given that we have > been leveraging it as part of our 2.7.4 release / upgrade process (in > addition to previous upgrades). > > The basic idea is to get full-scale black-box testing of the HDFS NN while > using significantly less (~10%) hardware than a real cluster of that size > would require. We use real NN images from our at-scale clusters paired with > some logic to fake out DNs into thinking they are storing data when they > are not, allowing us to stuff more DNs onto each machine. Since we use a > real image, we can replay real traces (collected from audit logs) to > compare actual production performance vs. performance on this simulated > cluster (with additional tuning, different version, etc.). We leverage YARN > to manage setting up this cluster and to replay the traces. > > Happy to answer questions. > > Erik > > On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko > wrote: > >> Hi Tianyi, >> >> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working >> on this project right now, I'll let him elaborate. >> Erik, you should probably respond on Apache dev list, as I think it could >> be interesting for other people as well, asince we planned to open source >> it. You can fork the "About 2.7.4 Release" thread with a new subject and >> give some details about Dynamometer there. >> >> Thanks, >> --Konstantin >> >> On Wed, Jul 19, 2017 at 1:40 AM, 何天一 wrote: >> >>> Hi, Shavachko. >>> >>> You mentioned an internal tool called Dynamometer to test NameNode >>> performance earlier in the 2.7.4 release thread. >>> I wonder if you could share some ideas behind the tool. Or is there a >>> plan to bring Dynamometer to open source community? >>> >>> Thanks. >>> >>> BR, >>> Tianyi >>> >>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko >>> wrote: >>> Hi everybody. We have been doing some internal testing of Hadoop 2.7.4. The testing is going well. Did not find any major issues on our workloads. Used an internal tool called Dynamometer to check NameNode performance on real cluster traces. Good. Overall test cluster performance looks good. Some more testing is still going on. I plan to build an RC next week. If there are no objection. Thanks, --Konst On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko < shv.had...@gmail.com> wrote: > Hey guys. > > An update on 2.7.4 progress. > We are down to 4 blockers. There is some work remaining on those. > https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814 > Would be good if people could follow up on review comments. > > I looked through nightly Jenkins build results for 2.7.4 both on Apache > Jenkins and internal. > Some test fail intermittently, but there no consistent failures. I filed > HDFS-11985 to track some of them. > https://issues.apache.org/jira/browse/HDFS-11985 > I do not currently consider these failures as blockers. LMK if some of > them are. > > We started internal testing of branch-2.7 on one of our smallish (100+ > nodes) test clusters. > Will update on the results. > > There is a plan to enable BigTop for 2.7.4 testing. > > Akira, Brahma thank you for setting up a wiki page for 2.7.4 release. > Thank you everybody for contributing to this effort. > > Regards, > --Konstantin > > > On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka > wrote: > >> Sure. >> If you want to edit the wiki, please tell me your ASF confluence account. >> >> -Akira >> >> On 2017/05/30 15:31, Rohith Sharma K S wrote: >> >>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will >>> solve RM HA unstability issues. >>>
Re: Apache Hadoop 2.8.2 Release Plan
+1 to base the 2.8.2 release off of the more recent activity on branch-2.8. Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that are in branch-2.8. There also are a lot of JIRAs that claim they are fixed in 2.8.2 but are not in branch-2.8.2. Having the 2.8.2 release be based on recent activity in branch-2.8 would solve both of these issues, and we'd only need to move the handful of JIRAs that have marked themselves correctly as fixed in 2.8.3 to be fixed in 2.8.2. Jason On Friday, July 21, 2017 10:01 AM, Kihwal Leewrote: Thanks for driving the next 2.8 release, Junping. While I was committing a blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but missing in branch-2.8.2. Perhaps it is safer and easier to simply rebranch 2.8.2. Thanks,Kihwal On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du wrote: Hi all, Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get released today which is a special security release. Now, we should work towards 2.8.2 release which aim for production deployment. The focus obviously is to fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We currently have 13 blocker/critical issues, and 10 of them are Patch Available. I plan to cut an RC in a month - target for releasing before end of Aug., to give enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won't fit the timeline. For progress of releasing effort, please refer our release wiki [2]. Please share thoughts if you have any. Thanks! Thanks, Junping [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x [2] 2.8 Release wiki: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release From: Vinod Kumar Vavilapalli Sent: Thursday, July 20, 2017 1:05 PM To: gene...@hadoop.apache.org Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released Hi all, The Apache Hadoop PMC has released version 2.8.1. You can get it from this page: http://hadoop.apache.org/releases.html#Download This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1. Please note that 2.8.x release line continues to be not yet ready for production use. Critical issues are being ironed out via testing and downstream adoption. Production users should wait for a subsequent release in the 2.8.x line. Thanks +Vinod - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: Apache Hadoop 2.8.2 Release Plan
Thanks for driving the next 2.8 release, Junping. While I was committing a blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but missing in branch-2.8.2. Perhaps it is safer and easier to simply rebranch 2.8.2. Thanks,Kihwal On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Duwrote: Hi all, Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get released today which is a special security release. Now, we should work towards 2.8.2 release which aim for production deployment. The focus obviously is to fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We currently have 13 blocker/critical issues, and 10 of them are Patch Available. I plan to cut an RC in a month - target for releasing before end of Aug., to give enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won't fit the timeline. For progress of releasing effort, please refer our release wiki [2]. Please share thoughts if you have any. Thanks! Thanks, Junping [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x [2] 2.8 Release wiki: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release From: Vinod Kumar Vavilapalli Sent: Thursday, July 20, 2017 1:05 PM To: gene...@hadoop.apache.org Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released Hi all, The Apache Hadoop PMC has released version 2.8.1. You can get it from this page: http://hadoop.apache.org/releases.html#Download This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1. Please note that 2.8.x release line continues to be not yet ready for production use. Critical issues are being ironed out via testing and downstream adoption. Production users should wait for a subsequent release in the 2.8.x line. Thanks +Vinod - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
Mukul Kumar Singh created HADOOP-14674: -- Summary: Correct javadoc for getRandomizedTempPath Key: HADOOP-14674 URL: https://issues.apache.org/jira/browse/HADOOP-14674 Project: Hadoop Common Issue Type: Bug Components: common Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh getRandomizedTempPath has incorrect javadoc where the javadoc specifies a parameter to the function however the function doesnt expects one. {code} /** * Get a temp path. This may or may not be relative; it depends on what the * {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path * under the relative path {@link #DEFAULT_TEST_DATA_PATH} * @param subpath sub path, with no leading "/" character * @return a string to use in paths */ public static String getRandomizedTempPath() { return getTempPath(RandomStringUtils.randomAlphanumeric(10)); } {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org